I’m Michael “Valentine” Smith. Cofounder & senior instructor at CFAR 2012–2018. I left the rationality scene 2018–2024 to do some soul-searching. I’ve now reentered the fray, hopefully a bit wiser but that’s not really mine to judge.
You can find my non-LW writing on my blog Relentless Dawn. You can also find my social media profiles via my Linktree (though I mostly post on Twitter, and basically not anywhere else).
this strikes me as an extraordinarily valuable social role, far beyond what mathematicians are usually granted.
Totally agree. I think mathematics is dramatically and tragically misunderstood.
I feel like mathematicians are being damned by faint praise here.
I don’t understand how. Mind clarifying?
I think the path here is to decouple human value from economic output.
I see two interpretations here. One I might agree with, the other I fairly strongly disagree with.
The one I might agree with: Economic output is often going to be better done by something nonhuman, so if human value is tied to output, we’re setting humanity up for replacement.
I say I might agree because it really depends on what we mean by “output”. In some sense, all signals are a kind of output. Even if we find some way of valuing human existence inherently, signs that something is human will be an output that gets valued in whatever we might want to call an economy.
The one I fairly strongly disagree with: We have to decouple human value from the economy. Humans have to be seen as valuable for their mere existence.
I think that can’t work. Human existence is a cost, even if negligible. An economy that does anything like optimization will have pressure in such a situation to wipe out humans to recover the cost. If the thing stopping the economy from doing that is some kind of rigid rule around “value”, then the economy will try to find some way to hack around that rule.
It seems to be that for humanity to survive this, human existence needs to be valued in a way that’s intrinsic to the economy.
Said differently, I think it needs to be on the table that what humanity most values and cherishes is best accomplished without humanity’s survival. If it can’t be, then the reason it can’t be needs to be what’s baked into the Great Optimizer (i.e. the thing that showed up as evolution, then the economy, now AI) in order for humanity to survive this juncture.
I’ve been thinking about this for a long time. Doesn’t mean my thoughts are right, just well worn. Hopefully deep in a truth-tracking direction.
My best guess is that mathematicians will need to make their job be about enriching subjective experience rather than generating proofs.
If humans’ value is defined by their automatable output, then the economy absolutely will replace humans. It seems to have maybe found a way to do that with AI.
If we think there’s something wrong with that, then we’d better start defining humans’ value in terms of whatever it is that gives us that sense of wrongness.
I don’t know exactly how we do that. But I I think I see at least part of what to do.
If we get that part right, I think maybe the economy stops actively maligning AI against human existence.
I really like this post. Thank you for writing it.
I want to briefly springboard off of this comment:
I’ve occasionally heard people say MIRI et al should stop visibly worrying about doom, lest it give AIs negative self-fulfilling predictions. I disagree; keeping one’s fears silent out of fear is not how one sets up long-term healthy friendships, AFAICT. (Though, speaking with some optimization toward “don’t needlessly hurt the other party’s feelings” is good, AFAICT.)
There’s a thing I often say that might be heard as or rounded to what you’re saying here. (Not to imply you’re thinking of me here. I’m just using your comment as an excuse to make a related point.) I want to try comparing/contrasting it.
The thing I’m trying to say is, I think folk should stop solely visibly worrying about doom. There needs to be “We’re really terrified about XYZ, and it could happen because of ABC… so let’s PQR instead.” Where PQR is about a positive vision, not just about a vision about stopping ABC or XYZ.
Like if I’m scared of getting an ulcer, and I focus on the risk factors of getting an ulcer, and I try to modify my diet patterns so that I decrease ulcer risk… there’s a way where I’m organizing all my attention around the threat of an ulcer. Everything becomes about it. At some point it becomes kind of self-defeating to keep focusing on reducing ulcer risk; I need to shift my attention to what positive vision for my life that having an ulcer would be a problem for. I want robust health and ability to eat what I want and enjoy it. There are steps in that direction that are less likely to arise from thinking in terms of ulcers but are likely to support reducing ulcer risk.
Or in navigating children or pets. A toddler who gets really fixated on my phone needs something else to organize her attention around. Just saying “No, you don’t get this phone” doesn’t address the issue. You have to offer her a positive alternative. Otherwise her attention just goes to the fact that you’re blockading her.
I am concerned about spelling out extremely detailed fears to the LLMs without any positive visions of what could happen instead. Or for those visions to be vague (e.g. “the glorious transhuman future”).
In the absence of such visions, I agree it’s important to still be able to express fears. But we’ve done that, in spades, and I think we need quite a bit more energy pointed at what we might like to see too.
(Notice that’s “too”, not “instead”. I’m opposing the “instead”, not the fears, and IMO that “instead” error is symmetric with respect to doom vs. hope.)
Oh absolutely. It’s an unwholesome attractor for me, and is part of why I try not to focus my time/attention on LW. I’ve gotten to where, whether accurate or not, my gut-level feeling is that LW kind of hates me and is hostile toward me, and I have to watch my step to be allowed to breathe around here. I try to hold a solid spine and not care about that hostility, both because I don’t want to create it if it’s not there and also because I think it’s unjust and I don’t want to succumb to bullies. But it’s tricky because the perceived vibe grabs me at an animal level.
E.g. it’s hard for me not to make stories about the fact that the comment you’re replying to right now has 3 karma and 4 votes. Meaning some are downvoting it. Why? I don’t know. Very hard not to make stories that feed the paranoid view.
Probably not harmless, but: I see myself in Rae’s post.
I actually have extremely passionate and direct things to say about the style of doomerism here. I think it’s deeply destructive and harmful and keeps creating damage unnecessarily, and the refusal to hear that message is selfish and a sign of serious mental illness at a collective level. I think there’s a better way, and it’s both more pleasant and a moral imperative to go that way instead. I’ve tried to speak about all this here many times, but I keep hitting mechanisms that Rae’s post extremely reminds me of.
E.g. I’m still not sorry for having written “Here’s the exit”. I suspect my saying so will cause this comment to get downvoted, and possibly have several people insisting I’m an Alex and not an Alice. And yet, as many haters as I’ve gotten for that post and for my lack of remorse for having written it, I’ve gotten roughly as many people DMing me or talking to me in person thanking me for it, some saying that it gave them their life back. I don’t like the hate, it hurts, but I’ll gladly take it if that’s the price of standing for what’s right.
If I were more skillful, I’d navigate the norms here in a respectful way that massages the truth into visibility. And I’m occasionally that skilled. But most of the time I’m just not. I can be gentle and polite, but then my points don’t seem to make a ripple. So I have to be kind of extreme and unreasonable to get the moral point heard here. Which enacts the Alice mechanism.
I mean, maybe there’s a more moderate way of expressing the problem I see that’s actually more accurate, but because of the kind of resistance I’ve run into, the only way for me to hold onto the thing I see is to be kind of crazy and extreme about it, which increases resistance. Commence loop.
I could give other examples from my interactions with the rationalist space. I seem to be particularly prone toward Alice-ness here. But hopefully the one example helps flesh out (what I see as) Rae’s message.
Thank you. And yes, that would be lovely to see.
Some folk on TPOT have been trying to compile positive visions of the future. I think there’s a book published of short stories of exactly that. In part meant to combat the hyperstitioning of doom by hyperstitioning something positive. I don’t recall the book’s name though.
I think the issue is that we’re building external intelligences instead of enhancing our own. We have a Cartesian relationship instead of an embedded one with respect to the current approach to AI (referencing Scott & Abram’s distinction of agent types). It’d be akin to a child building their adult form vs. the child growing into that adult.
I’d also add that we haven’t tried a Manhattan Project on aging. If we viewed ourselves as battling dark gods (which I think is a fair frame!) the way our grandparents were battling the Nazis, I think the motive would be there. And I think we would win. At this point maybe not before our parents die, and possibly not before the eldest among us die, but maybe we can bring them along anyway.
There are other puzzle we haven’t become wise enough to anticipate collectively. Curing aging leaves our institutions in shambles, because they’re largely built on assumptions of mortality and narrow reproductive windows. How do we prepare for a shift like that? I suspect that given the current way we do collective reasoning, the question won’t enter the Overton window until we have the equivalent of a magic pill curing aging and we’re watching the stock market and dating markets crash (or whatever).
One of the reasons I worked on building CFAR was that the species collectively just doesn’t seem to know how to stay focused on a problem that isn’t screaming an immediate threat of death in its face. That struck me as critically upstream of everything, including (but very much not limited to) sorting out AI risk. I think we’re capable of vastly more than we’ve been able to do so far, even without major advancements in tech, simply by getting the memetics right. The scientific revolution was IMO a preview. And thankfully, memetic revolutions can happen extremely quickly. (E.g. Japan became industrial in just one generation.)
For whatever reason, though, the death trance keeps having people being “practical” or “realistic” and giving up weirdly early.
I would describe what we have already done as radical life extension. Perhaps we have a difference in definition. From this link:
In 1900, the average life expectancy of a newborn was 32 years. By 2021 this had more than doubled to 71 years.
My understanding is that this phenomenon is mostly due to infant mortality. Like we get a bunch of dead zero-year-olds yanking the average way down. I’m given to understand that even in 1900, once someone hit 30 they were pretty likely to live for several more decades.
(I’ve not checked this carefully though. Reporting remembered impressions, not careful research.)
So Elua encourages variation and replication with weak selection, and Moloch encourages selection and variation with weak replication.
Nice. New thought for me. Thank you.
I sort of rotate the basis vectors a bit. I sometimes think of evolution as a dance between creativity (variation) and death (natural selection). In that spirit, I’m hearing you say that Elua encourages thriving via creativity, whereas Moloch encourages survival via death.
…Moloch is locally dominant when resources are scarce, and vice versa for Elua. […] (This is my best steelman of whatever the hell an “abundance mindset” is supposed to be)
I came to a similar steelman. Again with slightly shifted basis vectors though — basically the same ones as I mentioned up above I think. It comes out in terms of explore/exploit in my view. If you’re in a resource-poor situation, it’s high risk to explore, and you want to use whatever strategies you have on hand (with some exceptions, like if death is basically certain, at which point your strategies have already failed and you just want to increase variation a whole lot in a final survival bid). But if resources are abundant, then your long term survival is best served by basically preparing for forms of death that haven’t yet arrived. So expanding capacities via exploration. These seem to be two different modes (tight management of scarce resources vs. creative play to increase capacity in a high-resource domain).
If you’re actually in a high resource context but you can’t perceive it because your perceptions are contracted around a kind of emergency survival strategy, it’s helpful to “adopt an abundance mindset” so you can notice your context and correctly switch strategies.
If there’s some degree of self-fulfilling prophecy to the resources available (e.g. being confident you’ll get funds causes people to believe in your cause more and give you more funds), it’s also maybe helpful to assume you’re in an abundant context.
But if you are in fact in a scarce context and it’s not self-fulfilling, you very much want to budget and use what you know works.
I resonate so very strongly with this.
When I was five years old, it occurred to me that I might die someday. So I asked my parents if that was true. Mom kneeled down, held my shoulders, and said something like:
Oh, no, sweetheart. Everyone thinks everyone will die someday because they all get old. But we’ll fix aging before you get old. And if we’re wrong, we’ll just bring you back, because we love you.
It turned out my family was working on signing the whole family up for cryonics. They had books by the likes of Ettinger and Drexler filling their bookshelves. If I had a family religion, the glorious transhumanist timeline was it.
I’ve been finding myself wanting to talk more about this difference in general. I keep feeling like 21st century transhumanists don’t understand what the movement was. It was “trans” in the sense of “transcend and include”, not “go beyond and leave behind”. We were talking about things like reinforcing bones with diamond-like fiber placed there by nanobots, and delivering oxygen 20x more efficiently with little machines in our blood, so we’d be able to outrun modern cars and hold our breath at the bottom of a pool for an hour. And that was just the beginning — the simplest and most trivial applications of a growing mastery of reality.
I used to sign people up for cryonics with a degree of shared trust that if nothing else worked to stave off permanent death, this one would. There’s something utterly beautiful, that I have literally never seen in the rationality community, where a profound weight (what I came to call “the mantle of mortality”) would lift. People have no clue how heavily their expected eventual death weighs down on them. It’s truly beautiful to watch someone become free of it. It’s a type of relief folk have no comparison for as far as I can tell.
I took on that weight in October 2014. I’d held it and examined it when I was five, but I actually put the damn thing on last decade. Sort of an inverse karmic effect I suppose: I now vividly understand on the inside what subjective kindness I’d given others by its loss in myself.
I think the newer generations don’t understand what was lost with this doom-saturated approach to the future. It’s not just that AI could wipe us all out. It’s that hope and beauty are extremely hard to see from here, which makes focusing on something other than doom very, very hard. I experience it as a quality of building sandcastles on a sinkhole. Everything collapses eventually.
Having grown up a kind of immortal, there’s still a core of me that has unshakeable hope. And even with reflection today, that hope does not strike me as naïve. I remember at around age seven, when I came to understand what the deathist trance was, I vowed to myself:
If the answer ever appears to be that I should let myself die, just assume I’ve made an error in thinking. Death is never the answer.
By my read, folk raised mortal give up far, far too early. I don’t know what it is. It’s like despair is some kind of relief they want to collapse into. Like treating death as a kind of falling asleep, like it’s a rest, and maybe things will be better because Mom or Dad will fix it in the night. Or existence stops and that’s somehow relieving. I’m guessing; I don’t really understand the rationale. It looks to me like the individual human version of apoptosis.
Where I feel despair is at the level of collective action. Even now, I look around the LW-adjacent AI safety community with frustration and some exhaustion at the mortal sentiment that wants to express sorrow and grief and a kind of “Well, I guess we keep fighting against doom… but guys, I think we’ve lost.” It seems to me that this is the same apoptosis.
Death is never the answer. Grieving for our lost future before it’s determined strikes me as highly unwise.
But the death trance is far too strong, and I don’t see how to shake it, even in this subculture devoted to truth. The infection has successfully camouflaged itself as good epistemology. Reason, when pointed at the predicament, warps systematically to become arguments of hopelessness and despair. It was like that back in 2012, and possibly earlier, but it has metastasized and dominated the culture.
I have tried to push against this trance. But I don’t know what I’m doing, and I haven’t learned how to defeat it the way I’ve been trying, and now I’m tired.
I miss the innocence of anticipating the glorious future. Even calling it “transhumanist” feels strange, like a child talking about adulthood as “trans-child”. It once felt inevitable, and beautiful, and I watched as it became slowly more shared.
I dearly wish the culture here would loosen their fixation on “Don’t hit the doom tree!” and target something positive as an alternative. What does success look like? What vision can we call ourselves and humanity into?
There are yet still such visions. But first the collective needs to stop seeking to die. And I’ve lost faith that it will let its fixation go.
I miss the promise of the stars.
Ah yeah. Definitely a case of understatement on my part. My gut sense is more like “Obviously this is absolutely fucking critical and its neglect is one of the key things that’s been holding back real development of this art and the community that’s been trying to practice it for well over a decade.” But I think my stating it that way would tend to create more defensiveness than clarity, and we’d end up in a quagmire of argument that has more to do with feelings and social status than truth. The understated thing is at least true, and I think respects the cultural norms around shared epistemology better.
Now I’m less sure I disagree about anything.
That’s mostly my sense too. I think maybe we don’t disagree about anything substantial. Maybe not anything at all; mostly an aesthetic based on how we naturally view this idea cluster.
Control systems have to be a model of the system it’s controlling because of the good regulator theorem.
Oh man, this was exciting to discover was real. Makes some intuitive sense, but I sure do love when a plausibility argument (conjecture) becomes a necessity argument (proof).
…you agree that it’s possible to do truth tracking so that it works, even though “probably important” is a much weaker endorsement of this path than I’d have given.
I’m not sure what I said that you’re interpreting as my having said “probably important”.
To be clear, I think the core conjecture of LW-style rationality is that there’s a way to rearrange our problem-solving processes to be truth tracking in each, and maybe, every case; and that doing so is significantly more effective than the non-truth-tracking alternatives. If true, I think that’s way more significant than “probably important” would indicate.
In particular, there’s a plausible (though maybe unlikely) pathway here where getting this right could still outpace AI development via introducing a memetic “invasive species” and cause a tidal wave of sanity. It’s one of the few shots I still see as maybe working at this point if we’re in a fast takeoff scenario.
I’m sorry I’m being kind of thick here. Can you spell out plainly to me what exactly the problem your title is referring to is? “The friendly telepath problem” implies there’s some kind of problem that somehow involves friendly telepaths. You say I “kinda” described it correctly by saying it’s about how to create workable interfaces with friendly telepaths. Then you go on to say… um… I think to say that “the normally workable interface can still go wrong” as being the problem. Which I think is what I was saying? But you seem to say it as a correction to what I was trying to name.
So… what precisely is the friendly telepath problem?
I also appreciate how it engages with Chesterton’s Fence and suggests e.g. “For instance, the closeted homophobe should probably move out of his homophobic social context if he can.”
I’m glad you appreciate it… but I’m honestly unsure how this is a Chesterton’s Fence thing. Can you explain? You mean e.g. I’m not proposing the closeted homophobe just force himself to become aware of his sexual orientation without getting out of dodge?
(FWIW: if that’s what you meant, then I’m accidentally applying Chesterton’s Fence! It’s more like, I think I see the reason behind the pattern, and if correct then he has to address the reason behind the pattern in order for the pattern to shift. The theory kind of states that he can’t knock down Chesterton’s Fence here.)
I wonder this post is an infohazard for people immersed in sufficiently strong social incentives. I know you acknowledge this, but still.. Have you tested these tools with people in very difficult socially contexts?
Not extensively. A little bit.
FWIW I kind of lean toward the camp of tongue-in-cheek thinking that the idea of infohazards is one of the few real examples we have of infohazards. I think a more helpful framing is that memes interact in complicated ways with memetic ecosystems (the way any organism interacts in complicated ways with ecosystems), and it’s possible to have an “invasive species” kind of effect from a meme, and some of those effects can be pretty bad for the ecosystem in question. And I view a human psyche as a kind of memetic ecosystem. So it’s more a matter of, what’s the memetic impact? How does the ecosystem adapt? Does it become more vibrant, or less? How shall we measure that vibrance?
I think that point adds relevant nuance.
My impression is that a policy of worrying about infohazards encourages a memetically unhealthy environment. It seems like care, but in practice what I find actually happens is that critical information gets blocked, the policy becomes a vector for status games (e.g. taking a patronizing view of others, like you know better than they do what’s good or bad for them), and the memetic adaptation that would organically happen in contact with the truth just… doesn’t happen. So you end up with lots of bizarre fictions embedded in the memetic context in question.
Like, I don’t know, it sure doesn’t seem to me like what originally happened around Roko’s Basilisk was all that sane. The original idea strikes me as, at most, a self-fulfilling prophecy. It encourages the thinker to embed themselves in the idea and immerse themselves in an existential horror. And sure, you can do that if you want. And sure, knowing that you could makes it more likely. But I dunno, is knowing about crystal meth a basilisk? I think Roko’s Basilisk is better thought of as a training opportunity: learn how to not negotiate with terrorist memes. How do you handle self-fulfilling prophecies without bricking your ability to prioritize truth?
There was a whole section of this essay I cut that’s exactly about this topic. I suspect that conscious hyperstitioning, and learning to switch which attractor one hyperstitions in a multistable space, is also a key rationality skill. You need it in order to make truth-seeking safe to do when seeing a bad possibility threatens to make that possibility more likely. Otherwise it becomes strategic to refuse to acknowledge certain possibilities. That skill helps navigate situations where you’re embedded and you either don’t notice or can’t disembed (which is the strategy the OP focuses on: kind of disembedding the conscious mind).
Anyway. I ramble a bit. My point is that I don’t think worrying about infohazards is a good general approach.
That said, there’s a question of whether introducing these ideas to someone in a great deal of social stress (e.g. someone in an intensely psychologically abusive relationship, like with a full-on narcissist who’s really good at reading people) would make their situation worse.
I haven’t tested that very much. It’s possible. It strikes me as quite unlikely. I think most such people just… won’t engage with the ideas.
But sure, strong psychotechnologies can have strong effects, some of which might be negative (possibly severely negative) on some people.
There was a very early version of this material that I ran on a group back in 2019. It didn’t have the “Chesterton’s Fence” aspects to it basically at all. It was more like a frontal assault on self-deception. It created some intense subjective experiences for many (most?) of the people in that fairly large audience. Most such effects were extremely positive over the following year or so. But one person had a reactivation of her suicidal tendencies that left her spinning for months; she eventually pulled out of it, but mostly by distracting herself from the possibility of using that early version of these tools.
(The early version was almost nothing like the ones I’ve presented in the OP by the way. I don’t think the OP’s tools are at all easy to misuse the way that the 2019 version was almost trivial to misuse.)
The thing is, even that one woman who became suicidal again had extra factors that this post just doesn’t have. She was embedded in a social context that pressured her for weeks after that lecture to use the tools. That’s when she became suicidal. And she stopped being suicidal when everyone backed off and she went back to distracting herself.
So I think the concern is more like, let’s not start a cult where we pressure each other to eliminate all of our protected problems. A supportive context that doesn’t feed social control disorders but nonetheless respects and welcomes them despite their protected problems seems maybe extremely good. But trying to challenge one another to stop rationalizing in specific ways strikes me as setting up terrible incentives.
But I don’t know. What would you like to see in terms of adequate care? I want things to be good for people, but I also don’t want fear to prevent us from developing potent psychotechnologies.
One time I tried to help an online friend with her fear of needles, and it didn’t really work. Then she flew out and visited for a week, and the issue resolved itself without ever addressing it further. My best attempt to track back what happened is we did scarier stuff for fun (like jump rock) and she got to experience trusting someone else more than her own fears.
Recently, a different friend told me her Raynaud’s issue dissolved as soon as she saw me enter the room, because she was gonna ask for help and then realized that she didn’t actually need help because I was just gonna say “So stop?” and that she would. The active ingredient here being that I actually knew she could, and beliefs are contagious when well grounded and embodied.
Extremely cool. Yes. I’d love to see more things like these happening organically.
I get that your point here is that the entangling distorts things, but I worry it comes off like the problem is inherent in him being part of the system. “Truth seeking is actually worse” is a local property of his broken truth seeking strategy, not one inherent to being in the system.
I want to agree and disagree with you at the same time!
I agree that there’s a way for him to rearrange his relationship to the situation such that truth-seeking isn’t bad anymore. Strong agree. I think developing that type of rearrangement is probably an important part of a mature art of rationality.
But I do think the problem really is that he’s embedded in the system. I read your point as being that he doesn’t have to be. Yep, totally. But he is, and that’s the source of the problem. If he weren’t, then truth-seeking would be the strategic approach!
(Not to imply that being embedded always means truth-seeking is anti-strategic. Just that it can be and quite often is, and that that never happens AFAIK in non-embedded situations.)
I like your point here:
The problem isn’t with his embeddedness in the “him+her” or “him+her+others” system, it’s that he’s embedded in his own mind and hasn’t yet figured out what to do with “she might not like me”.
I’m not sure whether I agree. Maybe someone standing on the outside of the whole system might agree that yeah, the guy isn’t facing an embedded problem with the girl; he’s facing it with his thoughts about the girl instead. Seems abstractly plausible.
The thing is, on the inside that’s not at all how it looks to him. It looks like his worries are entangled with (his perception/projections of) her judgments of him.
And I’m not sure he’s totally wrong! If he gets tied up in knots about his own worry, that can be offputting to her. Which is what he’s worried about! Which is to say, his worry (or at least his meta-worry) is embedded with her perceptions of him, and therefore is part of the problem he’s trying to solve.
The key thing that I think causes truth-seeking to break down as a strategy for embedded problems is, your thinking and models can be part of the problem space rather than being objectively about the problem space. Which means that if having a distorted model helps solve the embedded problem, you’ll probably have a distorted model of what the problem is and/or how you’re solving it. So as long as the act of modeling the situation interacts directly with the situation, your models are suspect, even if you’re sincerely consciously trying to prioritize having true beliefs.
I think that issue is there even if the person is unknowingly objectively wrong about whether they’re embedded with the actual thing they’re trying to deal with.
If I’m reading you right, one of your arguments is that we need opacity with respect to ourselves so that we can’t edit ourselves, which then allows us to have rigid commitments, which is good for coordination.
I want to push back against that argument a bit. (Apologies if that’s not your argument! In which case I’ll be over here punching a tangential practice dummy.)
It does seem very plausible to me that rigidity in commitments is key for coordination. I hadn’t had that thought before reading your article.
It also seems plausible to me that one way of making one’s commitments rigid is by being unable to consciously access them and/or to make them “read only” so to speak.
But I think that last strategy is just one possibility, and it incentivizes irrationality.
Here’s another one: I know I need to maintain a commitment rigidly in order to coordinate with others. So I have a commitment which in some abstract sense I could edit, but I don’t, at a policy level.
This point is akin to how in principle my roommate could kill me. But I know he won’t. Not because he’s literally incapable of it, and not because it happens to be convenient for his local incentives, but because he’s attending to more long-range incentives. Even if he could become very rich by killing me, I’m pretty darn confident he wouldn’t.
The difference here is that instead of it being some rigid rule that we pretend organic strategic critters can’t edit in themselves, it’s a lucid recognition of the actual incentives involved, including the fact that having something playing the role of rigid rules is useful.
I think that kind of setup makes more thoughts thinkable, which avoids making irrationality strategic.
That said, I do think it’s helpful to have a barrier between your social interfaces and your conscious processing, and I do think it’s helpful for coordination. It’s just that the mechanism I see goes through the fact that having that barrier lets you think more strategically. With enough clarity, you can rederive why prosocial commitment devices make sense, and integrate that truth into your strategies (instead of having a bunch of potentially useful thoughts being unthinkable in invisible ways).
At least that’s how I’m currently approaching it.
All that said, I like your analysis for highlighting a thing I think in fact happens! Even if it should turn out not to be the optimal strategy for rationalists (which it still might be!).
Just to make sure I understand your idea right: the friendly telepath problem is something like “How do I create a workable interface with someone who might be able to collaborate with me, given that I’m kind of transparent to that person’s ‘telepathy’?” Is that basically right?
And then you’re suggesting a bunch of quirks about how to solve it, like how maintaining opaque parts of your psyche is actually helpful for solving this problem. Yes?
I wonder if the right term here is “persona” or “personality”. The mask your self wears to be a person in a social setting.
Roles strike me as different. I think of those as being about where you are and which character you’re playing in the social web. Your persona is more like how you’re playing whichever role you have.
Technically it’s a maybe. But my best guess is that this is mistaken. If you’re in WWII, you don’t want to grieve the loss of the world where you’ve defeated the Nazis. That saps your will to fight. Grief is for when hope is in fact lost. IMO.