I’m Michael “Valentine” Smith. Cofounder & senior instructor at CFAR 2012–2018. I left the rationality scene 2018–2024 to do some soul-searching. I’ve now reentered the fray, hopefully a bit wiser but that’s not really mine to judge.
You can find my non-LW writing on my Substack. You can also find my social media profiles via my Linktree (though I mostly post on Twitter, and basically not anywhere else).
I think the issue is that we’re building external intelligences instead of enhancing our own. We have a Cartesian relationship instead of an embedded one with respect to the current approach to AI (referencing Scott & Abram’s distinction of agent types). It’d be akin to a child building their adult form vs. the child growing into that adult.
I’d also add that we haven’t tried a Manhattan Project on aging. If we viewed ourselves as battling dark gods (which I think is a fair frame!) the way our grandparents were battling the Nazis, I think the motive would be there. And I think we would win. At this point maybe not before our parents die, and possibly not before the eldest among us die, but maybe we can bring them along anyway.
There are other puzzle we haven’t become wise enough to anticipate collectively. Curing aging leaves our institutions in shambles, because they’re largely built on assumptions of mortality and narrow reproductive windows. How do we prepare for a shift like that? I suspect that given the current way we do collective reasoning, the question won’t enter the Overton window until we have the equivalent of a magic pill curing aging and we’re watching the stock market and dating markets crash (or whatever).
One of the reasons I worked on building CFAR was that the species collectively just doesn’t seem to know how to stay focused on a problem that isn’t screaming an immediate threat of death in its face. That struck me as critically upstream of everything, including (but very much not limited to) sorting out AI risk. I think we’re capable of vastly more than we’ve been able to do so far, even without major advancements in tech, simply by getting the memetics right. The scientific revolution was IMO a preview. And thankfully, memetic revolutions can happen extremely quickly. (E.g. Japan became industrial in just one generation.)
For whatever reason, though, the death trance keeps having people being “practical” or “realistic” and giving up weirdly early.
I would describe what we have already done as radical life extension. Perhaps we have a difference in definition. From this link:
In 1900, the average life expectancy of a newborn was 32 years. By 2021 this had more than doubled to 71 years.
My understanding is that this phenomenon is mostly due to infant mortality. Like we get a bunch of dead zero-year-olds yanking the average way down. I’m given to understand that even in 1900, once someone hit 30 they were pretty likely to live for several more decades.
(I’ve not checked this carefully though. Reporting remembered impressions, not careful research.)
So Elua encourages variation and replication with weak selection, and Moloch encourages selection and variation with weak replication.
Nice. New thought for me. Thank you.
I sort of rotate the basis vectors a bit. I sometimes think of evolution as a dance between creativity (variation) and death (natural selection). In that spirit, I’m hearing you say that Elua encourages thriving via creativity, whereas Moloch encourages survival via death.
…Moloch is locally dominant when resources are scarce, and vice versa for Elua. […] (This is my best steelman of whatever the hell an “abundance mindset” is supposed to be)
I came to a similar steelman. Again with slightly shifted basis vectors though — basically the same ones as I mentioned up above I think. It comes out in terms of explore/exploit in my view. If you’re in a resource-poor situation, it’s high risk to explore, and you want to use whatever strategies you have on hand (with some exceptions, like if death is basically certain, at which point your strategies have already failed and you just want to increase variation a whole lot in a final survival bid). But if resources are abundant, then your long term survival is best served by basically preparing for forms of death that haven’t yet arrived. So expanding capacities via exploration. These seem to be two different modes (tight management of scarce resources vs. creative play to increase capacity in a high-resource domain).
If you’re actually in a high resource context but you can’t perceive it because your perceptions are contracted around a kind of emergency survival strategy, it’s helpful to “adopt an abundance mindset” so you can notice your context and correctly switch strategies.
If there’s some degree of self-fulfilling prophecy to the resources available (e.g. being confident you’ll get funds causes people to believe in your cause more and give you more funds), it’s also maybe helpful to assume you’re in an abundant context.
But if you are in fact in a scarce context and it’s not self-fulfilling, you very much want to budget and use what you know works.
I resonate so very strongly with this.
When I was five years old, it occurred to me that I might die someday. So I asked my parents if that was true. Mom kneeled down, held my shoulders, and said something like:
Oh, no, sweetheart. Everyone thinks everyone will die someday because they all get old. But we’ll fix aging before you get old. And if we’re wrong, we’ll just bring you back, because we love you.
It turned out my family was working on signing the whole family up for cryonics. They had books by the likes of Ettinger and Drexler filling their bookshelves. If I had a family religion, the glorious transhumanist timeline was it.
I’ve been finding myself wanting to talk more about this difference in general. I keep feeling like 21st century transhumanists don’t understand what the movement was. It was “trans” in the sense of “transcend and include”, not “go beyond and leave behind”. We were talking about things like reinforcing bones with diamond-like fiber placed there by nanobots, and delivering oxygen 20x more efficiently with little machines in our blood, so we’d be able to outrun modern cars and hold our breath at the bottom of a pool for an hour. And that was just the beginning — the simplest and most trivial applications of a growing mastery of reality.
I used to sign people up for cryonics with a degree of shared trust that if nothing else worked to stave off permanent death, this one would. There’s something utterly beautiful, that I have literally never seen in the rationality community, where a profound weight (what I came to call “the mantle of mortality”) would lift. People have no clue how heavily their expected eventual death weighs down on them. It’s truly beautiful to watch someone become free of it. It’s a type of relief folk have no comparison for as far as I can tell.
I took on that weight in October 2014. I’d held it and examined it when I was five, but I actually put the damn thing on last decade. Sort of an inverse karmic effect I suppose: I now vividly understand on the inside what subjective kindness I’d given others by its loss in myself.
I think the newer generations don’t understand what was lost with this doom-saturated approach to the future. It’s not just that AI could wipe us all out. It’s that hope and beauty are extremely hard to see from here, which makes focusing on something other than doom very, very hard. I experience it as a quality of building sandcastles on a sinkhole. Everything collapses eventually.
Having grown up a kind of immortal, there’s still a core of me that has unshakeable hope. And even with reflection today, that hope does not strike me as naïve. I remember at around age seven, when I came to understand what the deathist trance was, I vowed to myself:
If the answer ever appears to be that I should let myself die, just assume I’ve made an error in thinking. Death is never the answer.
By my read, folk raised mortal give up far, far too early. I don’t know what it is. It’s like despair is some kind of relief they want to collapse into. Like treating death as a kind of falling asleep, like it’s a rest, and maybe things will be better because Mom or Dad will fix it in the night. Or existence stops and that’s somehow relieving. I’m guessing; I don’t really understand the rationale. It looks to me like the individual human version of apoptosis.
Where I feel despair is at the level of collective action. Even now, I look around the LW-adjacent AI safety community with frustration and some exhaustion at the mortal sentiment that wants to express sorrow and grief and a kind of “Well, I guess we keep fighting against doom… but guys, I think we’ve lost.” It seems to me that this is the same apoptosis.
Death is never the answer. Grieving for our lost future before it’s determined strikes me as highly unwise.
But the death trance is far too strong, and I don’t see how to shake it, even in this subculture devoted to truth. The infection has successfully camouflaged itself as good epistemology. Reason, when pointed at the predicament, warps systematically to become arguments of hopelessness and despair. It was like that back in 2012, and possibly earlier, but it has metastasized and dominated the culture.
I have tried to push against this trance. But I don’t know what I’m doing, and I haven’t learned how to defeat it the way I’ve been trying, and now I’m tired.
I miss the innocence of anticipating the glorious future. Even calling it “transhumanist” feels strange, like a child talking about adulthood as “trans-child”. It once felt inevitable, and beautiful, and I watched as it became slowly more shared.
I dearly wish the culture here would loosen their fixation on “Don’t hit the doom tree!” and target something positive as an alternative. What does success look like? What vision can we call ourselves and humanity into?
There are yet still such visions. But first the collective needs to stop seeking to die. And I’ve lost faith that it will let its fixation go.
I miss the promise of the stars.
Ah yeah. Definitely a case of understatement on my part. My gut sense is more like “Obviously this is absolutely fucking critical and its neglect is one of the key things that’s been holding back real development of this art and the community that’s been trying to practice it for well over a decade.” But I think my stating it that way would tend to create more defensiveness than clarity, and we’d end up in a quagmire of argument that has more to do with feelings and social status than truth. The understated thing is at least true, and I think respects the cultural norms around shared epistemology better.
Now I’m less sure I disagree about anything.
That’s mostly my sense too. I think maybe we don’t disagree about anything substantial. Maybe not anything at all; mostly an aesthetic based on how we naturally view this idea cluster.
Control systems have to be a model of the system it’s controlling because of the good regulator theorem.
Oh man, this was exciting to discover was real. Makes some intuitive sense, but I sure do love when a plausibility argument (conjecture) becomes a necessity argument (proof).
…you agree that it’s possible to do truth tracking so that it works, even though “probably important” is a much weaker endorsement of this path than I’d have given.
I’m not sure what I said that you’re interpreting as my having said “probably important”.
To be clear, I think the core conjecture of LW-style rationality is that there’s a way to rearrange our problem-solving processes to be truth tracking in each, and maybe, every case; and that doing so is significantly more effective than the non-truth-tracking alternatives. If true, I think that’s way more significant than “probably important” would indicate.
In particular, there’s a plausible (though maybe unlikely) pathway here where getting this right could still outpace AI development via introducing a memetic “invasive species” and cause a tidal wave of sanity. It’s one of the few shots I still see as maybe working at this point if we’re in a fast takeoff scenario.
I’m sorry I’m being kind of thick here. Can you spell out plainly to me what exactly the problem your title is referring to is? “The friendly telepath problem” implies there’s some kind of problem that somehow involves friendly telepaths. You say I “kinda” described it correctly by saying it’s about how to create workable interfaces with friendly telepaths. Then you go on to say… um… I think to say that “the normally workable interface can still go wrong” as being the problem. Which I think is what I was saying? But you seem to say it as a correction to what I was trying to name.
So… what precisely is the friendly telepath problem?
I also appreciate how it engages with Chesterton’s Fence and suggests e.g. “For instance, the closeted homophobe should probably move out of his homophobic social context if he can.”
I’m glad you appreciate it… but I’m honestly unsure how this is a Chesterton’s Fence thing. Can you explain? You mean e.g. I’m not proposing the closeted homophobe just force himself to become aware of his sexual orientation without getting out of dodge?
(FWIW: if that’s what you meant, then I’m accidentally applying Chesterton’s Fence! It’s more like, I think I see the reason behind the pattern, and if correct then he has to address the reason behind the pattern in order for the pattern to shift. The theory kind of states that he can’t knock down Chesterton’s Fence here.)
I wonder this post is an infohazard for people immersed in sufficiently strong social incentives. I know you acknowledge this, but still.. Have you tested these tools with people in very difficult socially contexts?
Not extensively. A little bit.
FWIW I kind of lean toward the camp of tongue-in-cheek thinking that the idea of infohazards is one of the few real examples we have of infohazards. I think a more helpful framing is that memes interact in complicated ways with memetic ecosystems (the way any organism interacts in complicated ways with ecosystems), and it’s possible to have an “invasive species” kind of effect from a meme, and some of those effects can be pretty bad for the ecosystem in question. And I view a human psyche as a kind of memetic ecosystem. So it’s more a matter of, what’s the memetic impact? How does the ecosystem adapt? Does it become more vibrant, or less? How shall we measure that vibrance?
I think that point adds relevant nuance.
My impression is that a policy of worrying about infohazards encourages a memetically unhealthy environment. It seems like care, but in practice what I find actually happens is that critical information gets blocked, the policy becomes a vector for status games (e.g. taking a patronizing view of others, like you know better than they do what’s good or bad for them), and the memetic adaptation that would organically happen in contact with the truth just… doesn’t happen. So you end up with lots of bizarre fictions embedded in the memetic context in question.
Like, I don’t know, it sure doesn’t seem to me like what originally happened around Roko’s Basilisk was all that sane. The original idea strikes me as, at most, a self-fulfilling prophecy. It encourages the thinker to embed themselves in the idea and immerse themselves in an existential horror. And sure, you can do that if you want. And sure, knowing that you could makes it more likely. But I dunno, is knowing about crystal meth a basilisk? I think Roko’s Basilisk is better thought of as a training opportunity: learn how to not negotiate with terrorist memes. How do you handle self-fulfilling prophecies without bricking your ability to prioritize truth?
There was a whole section of this essay I cut that’s exactly about this topic. I suspect that conscious hyperstitioning, and learning to switch which attractor one hyperstitions in a multistable space, is also a key rationality skill. You need it in order to make truth-seeking safe to do when seeing a bad possibility threatens to make that possibility more likely. Otherwise it becomes strategic to refuse to acknowledge certain possibilities. That skill helps navigate situations where you’re embedded and you either don’t notice or can’t disembed (which is the strategy the OP focuses on: kind of disembedding the conscious mind).
Anyway. I ramble a bit. My point is that I don’t think worrying about infohazards is a good general approach.
That said, there’s a question of whether introducing these ideas to someone in a great deal of social stress (e.g. someone in an intensely psychologically abusive relationship, like with a full-on narcissist who’s really good at reading people) would make their situation worse.
I haven’t tested that very much. It’s possible. It strikes me as quite unlikely. I think most such people just… won’t engage with the ideas.
But sure, strong psychotechnologies can have strong effects, some of which might be negative (possibly severely negative) on some people.
There was a very early version of this material that I ran on a group back in 2019. It didn’t have the “Chesterton’s Fence” aspects to it basically at all. It was more like a frontal assault on self-deception. It created some intense subjective experiences for many (most?) of the people in that fairly large audience. Most such effects were extremely positive over the following year or so. But one person had a reactivation of her suicidal tendencies that left her spinning for months; she eventually pulled out of it, but mostly by distracting herself from the possibility of using that early version of these tools.
(The early version was almost nothing like the ones I’ve presented in the OP by the way. I don’t think the OP’s tools are at all easy to misuse the way that the 2019 version was almost trivial to misuse.)
The thing is, even that one woman who became suicidal again had extra factors that this post just doesn’t have. She was embedded in a social context that pressured her for weeks after that lecture to use the tools. That’s when she became suicidal. And she stopped being suicidal when everyone backed off and she went back to distracting herself.
So I think the concern is more like, let’s not start a cult where we pressure each other to eliminate all of our protected problems. A supportive context that doesn’t feed social control disorders but nonetheless respects and welcomes them despite their protected problems seems maybe extremely good. But trying to challenge one another to stop rationalizing in specific ways strikes me as setting up terrible incentives.
But I don’t know. What would you like to see in terms of adequate care? I want things to be good for people, but I also don’t want fear to prevent us from developing potent psychotechnologies.
One time I tried to help an online friend with her fear of needles, and it didn’t really work. Then she flew out and visited for a week, and the issue resolved itself without ever addressing it further. My best attempt to track back what happened is we did scarier stuff for fun (like jump rock) and she got to experience trusting someone else more than her own fears.
Recently, a different friend told me her Raynaud’s issue dissolved as soon as she saw me enter the room, because she was gonna ask for help and then realized that she didn’t actually need help because I was just gonna say “So stop?” and that she would. The active ingredient here being that I actually knew she could, and beliefs are contagious when well grounded and embodied.
Extremely cool. Yes. I’d love to see more things like these happening organically.
I get that your point here is that the entangling distorts things, but I worry it comes off like the problem is inherent in him being part of the system. “Truth seeking is actually worse” is a local property of his broken truth seeking strategy, not one inherent to being in the system.
I want to agree and disagree with you at the same time!
I agree that there’s a way for him to rearrange his relationship to the situation such that truth-seeking isn’t bad anymore. Strong agree. I think developing that type of rearrangement is probably an important part of a mature art of rationality.
But I do think the problem really is that he’s embedded in the system. I read your point as being that he doesn’t have to be. Yep, totally. But he is, and that’s the source of the problem. If he weren’t, then truth-seeking would be the strategic approach!
(Not to imply that being embedded always means truth-seeking is anti-strategic. Just that it can be and quite often is, and that that never happens AFAIK in non-embedded situations.)
I like your point here:
The problem isn’t with his embeddedness in the “him+her” or “him+her+others” system, it’s that he’s embedded in his own mind and hasn’t yet figured out what to do with “she might not like me”.
I’m not sure whether I agree. Maybe someone standing on the outside of the whole system might agree that yeah, the guy isn’t facing an embedded problem with the girl; he’s facing it with his thoughts about the girl instead. Seems abstractly plausible.
The thing is, on the inside that’s not at all how it looks to him. It looks like his worries are entangled with (his perception/projections of) her judgments of him.
And I’m not sure he’s totally wrong! If he gets tied up in knots about his own worry, that can be offputting to her. Which is what he’s worried about! Which is to say, his worry (or at least his meta-worry) is embedded with her perceptions of him, and therefore is part of the problem he’s trying to solve.
The key thing that I think causes truth-seeking to break down as a strategy for embedded problems is, your thinking and models can be part of the problem space rather than being objectively about the problem space. Which means that if having a distorted model helps solve the embedded problem, you’ll probably have a distorted model of what the problem is and/or how you’re solving it. So as long as the act of modeling the situation interacts directly with the situation, your models are suspect, even if you’re sincerely consciously trying to prioritize having true beliefs.
I think that issue is there even if the person is unknowingly objectively wrong about whether they’re embedded with the actual thing they’re trying to deal with.
If I’m reading you right, one of your arguments is that we need opacity with respect to ourselves so that we can’t edit ourselves, which then allows us to have rigid commitments, which is good for coordination.
I want to push back against that argument a bit. (Apologies if that’s not your argument! In which case I’ll be over here punching a tangential practice dummy.)
It does seem very plausible to me that rigidity in commitments is key for coordination. I hadn’t had that thought before reading your article.
It also seems plausible to me that one way of making one’s commitments rigid is by being unable to consciously access them and/or to make them “read only” so to speak.
But I think that last strategy is just one possibility, and it incentivizes irrationality.
Here’s another one: I know I need to maintain a commitment rigidly in order to coordinate with others. So I have a commitment which in some abstract sense I could edit, but I don’t, at a policy level.
This point is akin to how in principle my roommate could kill me. But I know he won’t. Not because he’s literally incapable of it, and not because it happens to be convenient for his local incentives, but because he’s attending to more long-range incentives. Even if he could become very rich by killing me, I’m pretty darn confident he wouldn’t.
The difference here is that instead of it being some rigid rule that we pretend organic strategic critters can’t edit in themselves, it’s a lucid recognition of the actual incentives involved, including the fact that having something playing the role of rigid rules is useful.
I think that kind of setup makes more thoughts thinkable, which avoids making irrationality strategic.
That said, I do think it’s helpful to have a barrier between your social interfaces and your conscious processing, and I do think it’s helpful for coordination. It’s just that the mechanism I see goes through the fact that having that barrier lets you think more strategically. With enough clarity, you can rederive why prosocial commitment devices make sense, and integrate that truth into your strategies (instead of having a bunch of potentially useful thoughts being unthinkable in invisible ways).
At least that’s how I’m currently approaching it.
All that said, I like your analysis for highlighting a thing I think in fact happens! Even if it should turn out not to be the optimal strategy for rationalists (which it still might be!).
Just to make sure I understand your idea right: the friendly telepath problem is something like “How do I create a workable interface with someone who might be able to collaborate with me, given that I’m kind of transparent to that person’s ‘telepathy’?” Is that basically right?
And then you’re suggesting a bunch of quirks about how to solve it, like how maintaining opaque parts of your psyche is actually helpful for solving this problem. Yes?
I wonder if the right term here is “persona” or “personality”. The mask your self wears to be a person in a social setting.
Roles strike me as different. I think of those as being about where you are and which character you’re playing in the social web. Your persona is more like how you’re playing whichever role you have.
I’m pretty happy with this piece. I often look back at things I wrote a year or more prior and find I’d like to approach the whole thing very differently. But I don’t feel that way at all here. I often want to point people to it as-is, usually without any caveats.
(The part I’d most rewrite is the section on how to use the framework to sometimes loosen Newcomblike self-deception. I’ve guided more people through that process since writing this article, and I’ve learned a few things about what’s helpful for folk to hear. But even that part doesn’t need substantial rewriting. I just suspect, without having given it a ton of thought, that it could be made slightly more effective for readers.)
The basic theory still seems quite solid to me. That there’s such a thing as a hostile telepath problem, and minds seek solutions to it when they encounter one, and that at least one of those solutions is a kind of self-deception, and therefore dissolving that kind of self-deception often (always?) requires having a different solution to the hostile telepath problem at hand.
I use this framework all the time in my personal debugging. It’s also a basic component in my analyses of strange mental behavior.[1] I hear people referencing the idea fairly often now in ways that make good sense to me. So I think it’s a pretty solid contribution.
I’ve also heard that this article inspired some folk to start running experiments and playing with novel psychotechnology schemes. I haven’t heard much about the results of such experiments. But it’s pretty exciting and delightful to know that the idea is of a sort that has spurred that sort of tinkering.
Today I made a few small edits to the post. I think they’re quite minor. They’re based on some comments from shortly after the post first went live that I found myself just agreeing with:
I agree with Ben Pace’s point. I think “gaining independence” is just a clearer way of pointing at that solution type than is “having power”. So I changed the name of that solution type throughout the piece.
Ninety-Three corrected me when I said in a footnote that psychopathy was a cluster B personality disorder. I’ve removed reference to “cluster B” or “personality disorder” from that footnote, instead replacing it with the clarification I gave in reply to the correcting comment.
I’m pretty confident that those edits don’t change anything meaningful about the post.
Overall: I’m very glad to have written this, and to have published it. It feels a bit odd to give my thoughts about whether it should be in the “Best of 2024”, for social reasons. But if I kind of ignore that and pretend this piece were written by someone else (to the extent that I can honestly do something like that), my sense is that it’s a very solid contender, and the main question I’d raise is how it compares to some of the other candidate posts that it would supplant.
Or rather, while that’s true, there’s a generalization I use all the time that the hostile telepath framework is an instantiation of. Hostile telepathy is one kind of social problem that can make a certain type of irrationality make sense to do if you don’t have a better solution. There are other social problems that can make some irrationality strategic too. See for instance “Social Control Disorders”. The generalization I use is something like: “If there were a social benefit to this pattern, what might it be? How might I goal-factor here once I see the possible social benefit?”
EDIT: a third update: man, I really want someone to actually try to “solve psychology” by combining insights from neo-hippies with systematizing thinking. Maybe this shouldn’t be CFAR’s job—I guess it used to be Leverage’s job, and that got kinda complicated—but I really want it to happen, and I don’t think academia will manage it any time soon.
FWIW, in a way that’s totally unrelated to what CFAR is up to, I’m looking to do something very close to this. I think subjective science is real and doable, and I’m pretty excited to make it happen. Step two might be to mimic the history of chemistry. But step one is more like a “literature review” of what actual science has actually been done, and which things (e.g. in psychology) are maybe kind of close or promising but need adjustment to hug the query properly. That includes things like looking at what Leverage has done.
There’s a ton to draw from in the neo-hippies’ stuff for sure, but nearly all of it is what I’d tag as “pre-scientific subjective engineering” (analogous to bloodletting in medicine: sometimes actually effective, but memetically optimized to appear effective and relevant within a pre-scientific context). I view them as a source of inspiration. Kind of like how alchemy had a ton of really interesting results that were what was to be investigated to create chemistry, but the frameworks of alchemy were almost perfectly useless for developing a science of chemical reactions.
The mother is not hostile, she is closely aligned to her child.
I just replied to another reviewer about this point. In short: I agree, I think it’s worth noticing, and I also think the point is irrelevant. The question isn’t whether the mother truly is hostile vs. aligned with the child. The question is whether the child experiences threat from an apparent telepath.
This point is related to footnote 4. I think it’s unhelpful to ask whether the mother “actually is” a hostile telepath. Hostile telepathy is about a perception someone has of another. If you perceive someone (or something) as a hostile telepath, you need some solution to that problem. One possible solution is to discover that they are not, in fact, hostile. But if you don’t converge on that solution, you’ll need some other one.
As I mentioned to the other reviewer, it stands out to me that two people both zoomed in on the same objection. I’m not sure what’s going on there. Let me know if I’ve missed your point?
The math class example also reads false to me. I have a kid who loves math and hates math class, and this does not seem like a difficult distinction to make.
Of course! My impression is that many (most?) math students don’t get sucked into the Newcomblike self-deception pattern I was naming. But some do! You’re pointing out an example of it not happening. If I were claiming that this happens for all math students, your point would totally refute mine. And to the degree you thought I was making that claim, or it came across ambiguous about whether I was, I’m glad you brought it up! But my point wasn’t that all math students encounter this. It’s that some do. And I don’t think it’s super rare.
The AI implications follow naturally.
Yep. Key to why I worked on it to begin with. I’m glad you caught that and named it!
My guess is self-deception is incentivized so long as other deception is incentivized
I think you’re probably right.
Just to be clear, I wasn’t trying to lay out a theory of all self-deception. I was trying to lay out a theory of one cause of at least one kind of self-deception (namely Newcomblike self-deception). I noticed that the problem seemed like it should be real, and it has multiple solutions, Newcomblike self-deception being one of them. That setup had some interesting logic that panned out with some casual experimentation.
I kind of wonder what portion of self-deception is entirely about dealing with hostile telepaths. Is it 100%? Probably not. My intuitive impression is that it’s well over 50% though. But even if it’s just 10% I don’t think that affects the logic of the post whatsoever. It’s a suggestion that thus-and-such type of self-deception arises from solving a particular problem that has other possible solutions, not a theory that all self-deception comes from this mechanism.
Regarding the mother whose glasses broke as hostile telepath: I think a more charitable interpretation is she’s ineffective socializer.
Someone else brought this up too. I think you’re right. And perhaps I wrote that part in a misleading way!
That said, I don’t think your very accurate point affects the example whatsoever.
The question isn’t about what’s really going on. The question is, from the perspective of the child, is he dealing with a hostile telepath? Which is to say, is he dealing with someone (a) who seems to be able to read his internal states and (b) whom he doesn’t trust won’t make his life worse based on what she finds? If the answer is “yes”, he’s faced with a hostile telepath problem, for which he needs some kind of solution.
It really doesn’t matter whatsoever how badly that represents the mother’s subjective state or motivations. The child doesn’t have access to that. The child just knows that Mother is mad at him, is demanding that he “be sorry”, and is checking. It’s possible the mom isn’t even mad! Maybe she’s a perfect saint bringing pure love and care and understanding while gently guiding the child to the best of her ability. But if the child perceives the mother as a hostile telepath, then he needs a solution. Which Newcomblike self-deception is one such solution.
It seems relevant that another reviewer gave the same pushback though. I wonder if I’ve been unclear, or if I’m missing something. Let me know if it seems to you like I’ve missed your point.
Thank you. And yes, that would be lovely to see.
Some folk on TPOT have been trying to compile positive visions of the future. I think there’s a book published of short stories of exactly that. In part meant to combat the hyperstitioning of doom by hyperstitioning something positive. I don’t recall the book’s name though.