I’m Michael “Valentine” Smith. Cofounder & senior instructor at CFAR 2012–2018. I left the rationality scene 2018–2024 to do some soul-searching. I’ve now reentered the fray, hopefully a bit wiser but that’s not really mine to judge.
You can find my non-LW writing on my Substack. You can also find my social media profiles via my Linktree (though I mostly post on Twitter, and basically not anywhere else).
Now I’m less sure I disagree about anything.
That’s mostly my sense too. I think maybe we don’t disagree about anything substantial. Maybe not anything at all; mostly an aesthetic based on how we naturally view this idea cluster.
Control systems have to be a model of the system it’s controlling because of the good regulator theorem.
Oh man, this was exciting to discover was real. Makes some intuitive sense, but I sure do love when a plausibility argument (conjecture) becomes a necessity argument (proof).
…you agree that it’s possible to do truth tracking so that it works, even though “probably important” is a much weaker endorsement of this path than I’d have given.
I’m not sure what I said that you’re interpreting as my having said “probably important”.
To be clear, I think the core conjecture of LW-style rationality is that there’s a way to rearrange our problem-solving processes to be truth tracking in each, and maybe, every case; and that doing so is significantly more effective than the non-truth-tracking alternatives. If true, I think that’s way more significant than “probably important” would indicate.
In particular, there’s a plausible (though maybe unlikely) pathway here where getting this right could still outpace AI development via introducing a memetic “invasive species” and cause a tidal wave of sanity. It’s one of the few shots I still see as maybe working at this point if we’re in a fast takeoff scenario.
I’m sorry I’m being kind of thick here. Can you spell out plainly to me what exactly the problem your title is referring to is? “The friendly telepath problem” implies there’s some kind of problem that somehow involves friendly telepaths. You say I “kinda” described it correctly by saying it’s about how to create workable interfaces with friendly telepaths. Then you go on to say… um… I think to say that “the normally workable interface can still go wrong” as being the problem. Which I think is what I was saying? But you seem to say it as a correction to what I was trying to name.
So… what precisely is the friendly telepath problem?
I also appreciate how it engages with Chesterton’s Fence and suggests e.g. “For instance, the closeted homophobe should probably move out of his homophobic social context if he can.”
I’m glad you appreciate it… but I’m honestly unsure how this is a Chesterton’s Fence thing. Can you explain? You mean e.g. I’m not proposing the closeted homophobe just force himself to become aware of his sexual orientation without getting out of dodge?
(FWIW: if that’s what you meant, then I’m accidentally applying Chesterton’s Fence! It’s more like, I think I see the reason behind the pattern, and if correct then he has to address the reason behind the pattern in order for the pattern to shift. The theory kind of states that he can’t knock down Chesterton’s Fence here.)
I wonder this post is an infohazard for people immersed in sufficiently strong social incentives. I know you acknowledge this, but still.. Have you tested these tools with people in very difficult socially contexts?
Not extensively. A little bit.
FWIW I kind of lean toward the camp of tongue-in-cheek thinking that the idea of infohazards is one of the few real examples we have of infohazards. I think a more helpful framing is that memes interact in complicated ways with memetic ecosystems (the way any organism interacts in complicated ways with ecosystems), and it’s possible to have an “invasive species” kind of effect from a meme, and some of those effects can be pretty bad for the ecosystem in question. And I view a human psyche as a kind of memetic ecosystem. So it’s more a matter of, what’s the memetic impact? How does the ecosystem adapt? Does it become more vibrant, or less? How shall we measure that vibrance?
I think that point adds relevant nuance.
My impression is that a policy of worrying about infohazards encourages a memetically unhealthy environment. It seems like care, but in practice what I find actually happens is that critical information gets blocked, the policy becomes a vector for status games (e.g. taking a patronizing view of others, like you know better than they do what’s good or bad for them), and the memetic adaptation that would organically happen in contact with the truth just… doesn’t happen. So you end up with lots of bizarre fictions embedded in the memetic context in question.
Like, I don’t know, it sure doesn’t seem to me like what originally happened around Roko’s Basilisk was all that sane. The original idea strikes me as, at most, a self-fulfilling prophecy. It encourages the thinker to embed themselves in the idea and immerse themselves in an existential horror. And sure, you can do that if you want. And sure, knowing that you could makes it more likely. But I dunno, is knowing about crystal meth a basilisk? I think Roko’s Basilisk is better thought of as a training opportunity: learn how to not negotiate with terrorist memes. How do you handle self-fulfilling prophecies without bricking your ability to prioritize truth?
There was a whole section of this essay I cut that’s exactly about this topic. I suspect that conscious hyperstitioning, and learning to switch which attractor one hyperstitions in a multistable space, is also a key rationality skill. You need it in order to make truth-seeking safe to do when seeing a bad possibility threatens to make that possibility more likely. Otherwise it becomes strategic to refuse to acknowledge certain possibilities. That skill helps navigate situations where you’re embedded and you either don’t notice or can’t disembed (which is the strategy the OP focuses on: kind of disembedding the conscious mind).
Anyway. I ramble a bit. My point is that I don’t think worrying about infohazards is a good general approach.
That said, there’s a question of whether introducing these ideas to someone in a great deal of social stress (e.g. someone in an intensely psychologically abusive relationship, like with a full-on narcissist who’s really good at reading people) would make their situation worse.
I haven’t tested that very much. It’s possible. It strikes me as quite unlikely. I think most such people just… won’t engage with the ideas.
But sure, strong psychotechnologies can have strong effects, some of which might be negative (possibly severely negative) on some people.
There was a very early version of this material that I ran on a group back in 2019. It didn’t have the “Chesterton’s Fence” aspects to it basically at all. It was more like a frontal assault on self-deception. It created some intense subjective experiences for many (most?) of the people in that fairly large audience. Most such effects were extremely positive over the following year or so. But one person had a reactivation of her suicidal tendencies that left her spinning for months; she eventually pulled out of it, but mostly by distracting herself from the possibility of using that early version of these tools.
(The early version was almost nothing like the ones I’ve presented in the OP by the way. I don’t think the OP’s tools are at all easy to misuse the way that the 2019 version was almost trivial to misuse.)
The thing is, even that one woman who became suicidal again had extra factors that this post just doesn’t have. She was embedded in a social context that pressured her for weeks after that lecture to use the tools. That’s when she became suicidal. And she stopped being suicidal when everyone backed off and she went back to distracting herself.
So I think the concern is more like, let’s not start a cult where we pressure each other to eliminate all of our protected problems. A supportive context that doesn’t feed social control disorders but nonetheless respects and welcomes them despite their protected problems seems maybe extremely good. But trying to challenge one another to stop rationalizing in specific ways strikes me as setting up terrible incentives.
But I don’t know. What would you like to see in terms of adequate care? I want things to be good for people, but I also don’t want fear to prevent us from developing potent psychotechnologies.
One time I tried to help an online friend with her fear of needles, and it didn’t really work. Then she flew out and visited for a week, and the issue resolved itself without ever addressing it further. My best attempt to track back what happened is we did scarier stuff for fun (like jump rock) and she got to experience trusting someone else more than her own fears.
Recently, a different friend told me her Raynaud’s issue dissolved as soon as she saw me enter the room, because she was gonna ask for help and then realized that she didn’t actually need help because I was just gonna say “So stop?” and that she would. The active ingredient here being that I actually knew she could, and beliefs are contagious when well grounded and embodied.
Extremely cool. Yes. I’d love to see more things like these happening organically.
I get that your point here is that the entangling distorts things, but I worry it comes off like the problem is inherent in him being part of the system. “Truth seeking is actually worse” is a local property of his broken truth seeking strategy, not one inherent to being in the system.
I want to agree and disagree with you at the same time!
I agree that there’s a way for him to rearrange his relationship to the situation such that truth-seeking isn’t bad anymore. Strong agree. I think developing that type of rearrangement is probably an important part of a mature art of rationality.
But I do think the problem really is that he’s embedded in the system. I read your point as being that he doesn’t have to be. Yep, totally. But he is, and that’s the source of the problem. If he weren’t, then truth-seeking would be the strategic approach!
(Not to imply that being embedded always means truth-seeking is anti-strategic. Just that it can be and quite often is, and that that never happens AFAIK in non-embedded situations.)
I like your point here:
The problem isn’t with his embeddedness in the “him+her” or “him+her+others” system, it’s that he’s embedded in his own mind and hasn’t yet figured out what to do with “she might not like me”.
I’m not sure whether I agree. Maybe someone standing on the outside of the whole system might agree that yeah, the guy isn’t facing an embedded problem with the girl; he’s facing it with his thoughts about the girl instead. Seems abstractly plausible.
The thing is, on the inside that’s not at all how it looks to him. It looks like his worries are entangled with (his perception/projections of) her judgments of him.
And I’m not sure he’s totally wrong! If he gets tied up in knots about his own worry, that can be offputting to her. Which is what he’s worried about! Which is to say, his worry (or at least his meta-worry) is embedded with her perceptions of him, and therefore is part of the problem he’s trying to solve.
The key thing that I think causes truth-seeking to break down as a strategy for embedded problems is, your thinking and models can be part of the problem space rather than being objectively about the problem space. Which means that if having a distorted model helps solve the embedded problem, you’ll probably have a distorted model of what the problem is and/or how you’re solving it. So as long as the act of modeling the situation interacts directly with the situation, your models are suspect, even if you’re sincerely consciously trying to prioritize having true beliefs.
I think that issue is there even if the person is unknowingly objectively wrong about whether they’re embedded with the actual thing they’re trying to deal with.
If I’m reading you right, one of your arguments is that we need opacity with respect to ourselves so that we can’t edit ourselves, which then allows us to have rigid commitments, which is good for coordination.
I want to push back against that argument a bit. (Apologies if that’s not your argument! In which case I’ll be over here punching a tangential practice dummy.)
It does seem very plausible to me that rigidity in commitments is key for coordination. I hadn’t had that thought before reading your article.
It also seems plausible to me that one way of making one’s commitments rigid is by being unable to consciously access them and/or to make them “read only” so to speak.
But I think that last strategy is just one possibility, and it incentivizes irrationality.
Here’s another one: I know I need to maintain a commitment rigidly in order to coordinate with others. So I have a commitment which in some abstract sense I could edit, but I don’t, at a policy level.
This point is akin to how in principle my roommate could kill me. But I know he won’t. Not because he’s literally incapable of it, and not because it happens to be convenient for his local incentives, but because he’s attending to more long-range incentives. Even if he could become very rich by killing me, I’m pretty darn confident he wouldn’t.
The difference here is that instead of it being some rigid rule that we pretend organic strategic critters can’t edit in themselves, it’s a lucid recognition of the actual incentives involved, including the fact that having something playing the role of rigid rules is useful.
I think that kind of setup makes more thoughts thinkable, which avoids making irrationality strategic.
That said, I do think it’s helpful to have a barrier between your social interfaces and your conscious processing, and I do think it’s helpful for coordination. It’s just that the mechanism I see goes through the fact that having that barrier lets you think more strategically. With enough clarity, you can rederive why prosocial commitment devices make sense, and integrate that truth into your strategies (instead of having a bunch of potentially useful thoughts being unthinkable in invisible ways).
At least that’s how I’m currently approaching it.
All that said, I like your analysis for highlighting a thing I think in fact happens! Even if it should turn out not to be the optimal strategy for rationalists (which it still might be!).
Just to make sure I understand your idea right: the friendly telepath problem is something like “How do I create a workable interface with someone who might be able to collaborate with me, given that I’m kind of transparent to that person’s ‘telepathy’?” Is that basically right?
And then you’re suggesting a bunch of quirks about how to solve it, like how maintaining opaque parts of your psyche is actually helpful for solving this problem. Yes?
I wonder if the right term here is “persona” or “personality”. The mask your self wears to be a person in a social setting.
Roles strike me as different. I think of those as being about where you are and which character you’re playing in the social web. Your persona is more like how you’re playing whichever role you have.
I’m pretty happy with this piece. I often look back at things I wrote a year or more prior and find I’d like to approach the whole thing very differently. But I don’t feel that way at all here. I often want to point people to it as-is, usually without any caveats.
(The part I’d most rewrite is the section on how to use the framework to sometimes loosen Newcomblike self-deception. I’ve guided more people through that process since writing this article, and I’ve learned a few things about what’s helpful for folk to hear. But even that part doesn’t need substantial rewriting. I just suspect, without having given it a ton of thought, that it could be made slightly more effective for readers.)
The basic theory still seems quite solid to me. That there’s such a thing as a hostile telepath problem, and minds seek solutions to it when they encounter one, and that at least one of those solutions is a kind of self-deception, and therefore dissolving that kind of self-deception often (always?) requires having a different solution to the hostile telepath problem at hand.
I use this framework all the time in my personal debugging. It’s also a basic component in my analyses of strange mental behavior.[1] I hear people referencing the idea fairly often now in ways that make good sense to me. So I think it’s a pretty solid contribution.
I’ve also heard that this article inspired some folk to start running experiments and playing with novel psychotechnology schemes. I haven’t heard much about the results of such experiments. But it’s pretty exciting and delightful to know that the idea is of a sort that has spurred that sort of tinkering.
Today I made a few small edits to the post. I think they’re quite minor. They’re based on some comments from shortly after the post first went live that I found myself just agreeing with:
I agree with Ben Pace’s point. I think “gaining independence” is just a clearer way of pointing at that solution type than is “having power”. So I changed the name of that solution type throughout the piece.
Ninety-Three corrected me when I said in a footnote that psychopathy was a cluster B personality disorder. I’ve removed reference to “cluster B” or “personality disorder” from that footnote, instead replacing it with the clarification I gave in reply to the correcting comment.
I’m pretty confident that those edits don’t change anything meaningful about the post.
Overall: I’m very glad to have written this, and to have published it. It feels a bit odd to give my thoughts about whether it should be in the “Best of 2024”, for social reasons. But if I kind of ignore that and pretend this piece were written by someone else (to the extent that I can honestly do something like that), my sense is that it’s a very solid contender, and the main question I’d raise is how it compares to some of the other candidate posts that it would supplant.
Or rather, while that’s true, there’s a generalization I use all the time that the hostile telepath framework is an instantiation of. Hostile telepathy is one kind of social problem that can make a certain type of irrationality make sense to do if you don’t have a better solution. There are other social problems that can make some irrationality strategic too. See for instance “Social Control Disorders”. The generalization I use is something like: “If there were a social benefit to this pattern, what might it be? How might I goal-factor here once I see the possible social benefit?”
EDIT: a third update: man, I really want someone to actually try to “solve psychology” by combining insights from neo-hippies with systematizing thinking. Maybe this shouldn’t be CFAR’s job—I guess it used to be Leverage’s job, and that got kinda complicated—but I really want it to happen, and I don’t think academia will manage it any time soon.
FWIW, in a way that’s totally unrelated to what CFAR is up to, I’m looking to do something very close to this. I think subjective science is real and doable, and I’m pretty excited to make it happen. Step two might be to mimic the history of chemistry. But step one is more like a “literature review” of what actual science has actually been done, and which things (e.g. in psychology) are maybe kind of close or promising but need adjustment to hug the query properly. That includes things like looking at what Leverage has done.
There’s a ton to draw from in the neo-hippies’ stuff for sure, but nearly all of it is what I’d tag as “pre-scientific subjective engineering” (analogous to bloodletting in medicine: sometimes actually effective, but memetically optimized to appear effective and relevant within a pre-scientific context). I view them as a source of inspiration. Kind of like how alchemy had a ton of really interesting results that were what was to be investigated to create chemistry, but the frameworks of alchemy were almost perfectly useless for developing a science of chemical reactions.
The mother is not hostile, she is closely aligned to her child.
I just replied to another reviewer about this point. In short: I agree, I think it’s worth noticing, and I also think the point is irrelevant. The question isn’t whether the mother truly is hostile vs. aligned with the child. The question is whether the child experiences threat from an apparent telepath.
This point is related to footnote 4. I think it’s unhelpful to ask whether the mother “actually is” a hostile telepath. Hostile telepathy is about a perception someone has of another. If you perceive someone (or something) as a hostile telepath, you need some solution to that problem. One possible solution is to discover that they are not, in fact, hostile. But if you don’t converge on that solution, you’ll need some other one.
As I mentioned to the other reviewer, it stands out to me that two people both zoomed in on the same objection. I’m not sure what’s going on there. Let me know if I’ve missed your point?
The math class example also reads false to me. I have a kid who loves math and hates math class, and this does not seem like a difficult distinction to make.
Of course! My impression is that many (most?) math students don’t get sucked into the Newcomblike self-deception pattern I was naming. But some do! You’re pointing out an example of it not happening. If I were claiming that this happens for all math students, your point would totally refute mine. And to the degree you thought I was making that claim, or it came across ambiguous about whether I was, I’m glad you brought it up! But my point wasn’t that all math students encounter this. It’s that some do. And I don’t think it’s super rare.
The AI implications follow naturally.
Yep. Key to why I worked on it to begin with. I’m glad you caught that and named it!
My guess is self-deception is incentivized so long as other deception is incentivized
I think you’re probably right.
Just to be clear, I wasn’t trying to lay out a theory of all self-deception. I was trying to lay out a theory of one cause of at least one kind of self-deception (namely Newcomblike self-deception). I noticed that the problem seemed like it should be real, and it has multiple solutions, Newcomblike self-deception being one of them. That setup had some interesting logic that panned out with some casual experimentation.
I kind of wonder what portion of self-deception is entirely about dealing with hostile telepaths. Is it 100%? Probably not. My intuitive impression is that it’s well over 50% though. But even if it’s just 10% I don’t think that affects the logic of the post whatsoever. It’s a suggestion that thus-and-such type of self-deception arises from solving a particular problem that has other possible solutions, not a theory that all self-deception comes from this mechanism.
Regarding the mother whose glasses broke as hostile telepath: I think a more charitable interpretation is she’s ineffective socializer.
Someone else brought this up too. I think you’re right. And perhaps I wrote that part in a misleading way!
That said, I don’t think your very accurate point affects the example whatsoever.
The question isn’t about what’s really going on. The question is, from the perspective of the child, is he dealing with a hostile telepath? Which is to say, is he dealing with someone (a) who seems to be able to read his internal states and (b) whom he doesn’t trust won’t make his life worse based on what she finds? If the answer is “yes”, he’s faced with a hostile telepath problem, for which he needs some kind of solution.
It really doesn’t matter whatsoever how badly that represents the mother’s subjective state or motivations. The child doesn’t have access to that. The child just knows that Mother is mad at him, is demanding that he “be sorry”, and is checking. It’s possible the mom isn’t even mad! Maybe she’s a perfect saint bringing pure love and care and understanding while gently guiding the child to the best of her ability. But if the child perceives the mother as a hostile telepath, then he needs a solution. Which Newcomblike self-deception is one such solution.
It seems relevant that another reviewer gave the same pushback though. I wonder if I’ve been unclear, or if I’m missing something. Let me know if it seems to you like I’ve missed your point.
I highly doubt it is explanatory for the field and the associated risk predictions to exist in the first place, or that its validity should be questioned on such grounds, but this seems to happen in the article if I’m not entirely misreading it.
Not entirely. It’s a bit of a misreading. In this case I think the bit matters though.
(And it’s an understandable bit! It’s a subtle point I find I have a hard time communicating clearly.)
I’m trying to say two things:
There sure do seem to be some bad psychological influences going on.
It’s harder to tell what’s real when you have sufficiently bad psychological influences going on.
I think some people, such as you, are reacting really strongly to that second point. Like I’m taking a stand for AI risk being a non-issue and saying it’s all psychological projection.
I’m saying that nonzero, but close to zero. It’s a more plausible hypothesis to me than I think it is to this community. But that’s not because I’m going through the arguments that AI risk is real and finding refutations. It’s because I’ve seen some shockingly basic things turn out to be psychological projection, and I don’t think Less Wrong collectively understands that projection really can be that deep. I just don’t see it accounted for in the arguments for doom.
But that’s not the central point I’m trying to make. My point is more that I think the probability of doom is significantly elevated as a result of how memetic evolution works — and, stupidly, I think that makes doom more likely as a result of the “Don’t hit the tree” phenomenon.
And maybe even more centrally, you cannot know how elevated the probability is until you seriously check for memetic probability boosters. And even then, how you check needs to account for those memetic influences.
I’m not trying to say that AI safety shouldn’t exist as a field though.
From my point of view, there is already an overemphasis on psychological factors in the broader debate and it would be desirable to get back to the object level
Wow, you and I sure must be seeing different parts of the debate! I approximately only hear people talking about the object level. That’s part of my concern.
I mean, I see some folk doing hot takes on Twitter about psychological angles. But most of those strike me as more like pot shots and less like attempts to engage in a dialogue.
This was a great steelmanning, and is exactly the kind of thing I hope people will do in contact with what I offer. Even though I don’t agree with every detail, I feel received and like the thing I care about is being well enough held. Thank you.
Good call. I haven’t been reading Less Wrong in enough detail for a while to pull this up usefully. My impression comes from in-person conversations plus Twitter interactions. The thickest use of my encountering these terms in rationality circles was admittedly about a decade ago. But I’m not sure how much of that is due to my not spending as much time in rationality circles versus discourse norms moving on. I still encounter it almost solely from folk tied to LW-style rationality.
I don’t recall hearing you use the terms in ways that bothered me this way, FWIW.
it still seems bad to advocate for the exactly wrong policy, especially one that doesn’t make sense even if you turn out to be correct (as habryka points out in the original comment, many think 2028 is not really when most people expect agi to have happened).
I’m super sensitive to framing effects. I notice one here. I could be wrong, and I’m guessing that even if I’m right you didn’t intend it. But I want to push back against it here anyway. Framing effects don’t have to be intentional!
It’s not that I started with what I thought was a wrong or bad policy and tried to advocate for it. It’s that given all the constraints, I thought that preregistering a possibility as a “pause and reconsider” moment might be the most effective and respectful. It’s not what I’d have preferred if things were different. But things aren’t different from how they are, so I made a guess about the best compromise.
I then learned that I’d made some assumptions that weren’t right, and that determining such a pause point that would have collective weight is much more tricky. Alas.
But it was Oliver’s comment that brought this problem to my awareness. At no point did I advocate for what I thought at the time was the wrong policy. I had hope because I thought folk were laying down some timeline predictions that could be falsified soon. Turns out, approximately nope.
i think you would have much more luck advocating for chilling today and citing past evidence to make your case..
Empirically I disagree. That demonstrably has not been within the reach of my skill to do effectively. But it’s a sensible thing to consider trying again sometime.
Although I don’t like comments starting with “your logic slipped” because it sounds passive-aggressive “you are stupid” vibes I will reply.
Sorry, that’s not how I meant it. I meant it more like “Oh, I think your foot slipped there, so if you take another step I think it won’t have the effect you’re looking for.” We can all slip up. It’s intended as a friendly note.
I agree that on rereading it it didn’t come across that way.
So what you are saying is that yes this time is different just not today. It will definately happen and all the doomerism is correct but not on a short timeline because ____ insert reasoning that is different than what the top AI minds are saying today.
Uh, no. That’s not what I’m saying.
I’m saying something more like: if it turns out that doomerism is once again exaggerated, perhaps we should take a step back and ask what’s creating the exaggeration instead of plowing ahead as we have been.
I love this direction of inquiry. It’s tricky to get right because of lots of confounds. But I think something like it should be doable.
I love that this is a space where people care about this stuff.
I’m poorly suited to actually carry out these examinations on my own. But if someone does, or if someone wants to lead the charge and would like me to help design the study, I’d love to hear about it!
Ah yeah. Definitely a case of understatement on my part. My gut sense is more like “Obviously this is absolutely fucking critical and its neglect is one of the key things that’s been holding back real development of this art and the community that’s been trying to practice it for well over a decade.” But I think my stating it that way would tend to create more defensiveness than clarity, and we’d end up in a quagmire of argument that has more to do with feelings and social status than truth. The understated thing is at least true, and I think respects the cultural norms around shared epistemology better.