Communications lead at MIRI. Unless otherwise indicated, my posts and comments here reflect my own views, and not necessarily my employer’s.
Rob Bensinger
Karma: 21,197
Discussion with Eliezer Yudkowsky on AGI interventions
A challenge for AGI organizations, and a challenge for readers
Politics is way too meta
Biases: An Introduction
Elements of Rationalist Discourse
On saving one’s world
The basic reasons I expect AGI ruin
The inordinately slow spread of good AGI conversations in ML
ITT-passing and civility are good; “charity” is bad; steelmanning is niche
Outline of Galef’s “Scout Mindset”
Copying over a related Oct. 13-17 conversation from Facebook:
(context: someone posted a dating ad in a rationalist space where they said they like tarot etc., and rationalists objected)
_____________________________________________
Marie La: As a cultural side note, most of my woo knowledge (like how to read tarot) has come from the rationalist community, and I wouldn’t have learned it otherwise
_____________________________________________
Eliezer Yudkowsky: @Marie La Any ideas how we can stop that?
(+1 from Rob B)
_____________________________________________
Marie La: Idk, it’s an introspective technique that works for some people. Doesn’t particularly work for me. Sounds like the concern is bad optics / PR rather than efficacy
(+1 from Rob B)
_____________________________________________
Shaked Koplewitz: @Marie La optics implies that the concern is with the impression it makes on outsiders, my concern here is the effect on insiders (arguably this is optics too, but a non-central example)
_____________________________________________
Rob Bensinger: If the concern is optics, either to insiders or outsiders, then it seems vastly weaker to me than if the concern is about epistemic methods or conclusions. (Indeed, it might flip the sign for me.)
The rationality community should be about truth and winning, not about linking ourselves up to whatever is culturally associated with the word “rationality”.
The first-order argument for trying weird things and seeing if they work (or just doing them for fun as a sort of game, etc.) makes sense. I’d rather focus on the question of whether that first-order case fails epistemically and/or instrumentally. Also: what does it fail for? Just saying ‘woo’ doesn’t tell us whether, e.g., we should stop using IFS because it isn’t normal-sounding enough.
(+1 from Marie L)
_____________________________________________
Marie La: When we come across someone using weird mind trick X, we should figure out what it does and if we want the results. Being skilled at sorting out good weird mind tricks from bad, regardless of cultural coding, feels like an important rationalist skill.
Tarot is set of fancy art cards can be used in many ways, some that encourage magical thinking and some that provide useful introspective access
For the latter, I’d guess it’s somewhat useful to some people, similar to the skill of flipping a coin and doing what you want anyway
(+1 from Rob B)
_____________________________________________
Eliezer Yudkowsky: @Marie La I disagree and think the woo has proven in empirical practice to be sufficiently destructive to people who can’t see the destruction, to reach a level where it should not be tolerated by this group as a future subgroup norm, same as LSD use shouldn’t be tolerated by us as a subgroup norm.
(+1 from Rob B, Marie L)
_____________________________________________
Marie La: I’m interested in seeing more of your reasoning on this. Pointing out the harm model sounds useful to people who can’t easily see it (or to the people around them) to help avoid further harm in the future
(+1 from Rob B)
_____________________________________________
Eliezer Yudkowsky: The #1 reason why I think it’s harmful isn’t a theory by which I divined it in advance. Though there sure is a very obvious theory whereby the path of sanity is a narrow one and people who step a bit off it in what they fondly imagine to be a controlled way, fall quite a lot further once they’re hanging around with crazier people, crazier ideas, and have already decided to let themselves go a little.
The #1 reason why I think it’s harmful is the number of times you hear about somebody, or worse, some subgroup, that pushed a little woo on somebody, or offered them some psychedelics, and a few years later you’re hearing about how far they went off the deep end. It seems to be destructive in practice and that’s a far stronger reason to be wary than the obvious-seeming ways it could be destructive in principle.
(+1 from Anonymous, Rob B)
_____________________________________________
Anonymous: Agree with this but surely it also matters how often that action seems to have that effect *out of the times it’s done* - noting this because my impression (which however I don’t have data for) is that psychedelic use might be locally common enough that it’s only a small proportion of “rationalists who try LSD” who end up “going off the deep end”. Whereas experimenting with woo-y beliefs seems more strongly associated with that kind of trajectory and for that I endorse your conclusion.
(+1 from Eliezer Y)
_____________________________________________
Aella: @Eliezer Yudkowsky On phone so thumb words but I notice I have a belief that this is predictable, and thus not dangerous? or rather, it’s something like if you’re religious and noticed some ppl have been drinking alcohol and then eventually losing their faith, you might be right to be wary of alcohol, but if you know that it’s actually the doubt of their faith that *causes* the alcohol drinking, then you wouldn’t be concerned if someone drinks alcohol but also isn’t doubting their faith.
similarly I have some intuition here that the woo stuff is a symptom and not the cause, and that it’s very possible to engage harmlessly with the symptom alone, and is a fine social norm if people can distinguish the alcoholism from doubting your faith.
I do agree seeing woo belief does up my probability they might end up going off the deep end tho
(+1 from Rob B, Eliezer Y)
_____________________________________________
Eliezer Yudkowsky: My sense of “this seems to be ending very poorly on average” is much stronger for situations in which there’s a Leader or a Discernible Subgroup has formed, that are going up to others and saying “why, you really should try some psychedelics / woo”. Or where they wander up to individuals trying that, and put their arm around their shoulders all friendly-like.
Though I suppose that could also be because I’m much less likely to hear about the individual non-social cases even if they end poorly. And indeed, my sense of the individual cases, is that I have heard of a lot more individuals who took psychedelics a few times in a situation devoid of Leaders and Subgroups and nothing bad happened to them; compared to the case with woo, where it feels like I’m more likely to have heard that an individual who tried woo even in a situation without Leaders or Subgroups later went further off the deep end. Which has the very obvious explanation that some people ever do benefit from psychedelics and they are plausibly interesting to a healthy mind willing to risk itself, you can be sane and still try shrooms ever; while the woo thing requires a larger, more willing step off the strict rails of sanity.
But in the case of Subgroups or Leaders, neither woo nor psychedelics seems to end well.
And to be clear that this isn’t just disguised “boo subgroups and leaders”, let’s be clear that, say, OpenPhil is for these purposes a Subgroup and Holden Karnofsky is a Leader, as are MIRI and myself; they are just Subgroups and Leaders which have not, to my knowledge, ever advocated LSD or tarot readings.
(+1 from Rob B, Anonymous, Marie L)
_____________________________________________
Marie La: @Eliezer Yudkowsky Strong agree that psychedelics + a strong leader context can enable great harm quickly.
It tends to make people vulnerable, and if there’s a bad actor around this can be dangerous. This has been plenty weaponized for controlling people in cult-like groups.
I’d expand this to include most drugs, but especially classical psychs and mdma.
Here I’d blame the leader for seeking out vulnerable psychedelic users or encouraging people under them to use psychedelics, rather than the drugs themselves
(I can’t say much about weaponized woo, don’t know what that looks like as much)
(+1 from Anonymous)
_____________________________________________
Rob Bensinger: https://slatestarcodex.com/2019/09/10/ssc-journal-club-relaxed-beliefs-under-psychedelics-and-the-anarchic-brain suggests that psychedelics permanently “relax” people’s perceptual and epistemic priors, and that this is maybe why they can cause hallucinogen persisting perceptual disorder and crazy beliefs.
This seems maybe much worse for people whose starting priors are quite good. If your life is a wreck and you’re terrible at figuring out what’s true, then yeah, shaking things up might be great. But if rationalists are selected for having unusually *good* priors, then shaking things up will cause regression to the mean.
Cf. Jim Babcock’s argument that you shouldn’t be experimenting with new diets if you’re already an unusually awesome, productive, etc. person, since then you risk breaking a complex system that’s working well. It’s the people whose status quo sucks who should be experimenting.
(+1 from Jim B, Eliezer Y, Anonymous, Marie L)
_____________________________________________
Jim Babcock: My own impression is that the effect of LSD is not primarily a regression to the mean thing, but rather, that it temporarily enables some self-modification capabilities, which can be powerfully positive but which require a high degree of sanity and care to operate safely. When I see other people using psychedelics, very often I see them acting like Harry Potter experimenting with transfiguration, or worse, treating it as *entertainment*. And I want to yell at them, and point them at the scene in Dumbledore’s laboratory where Dumbledore and Minerva go through a checklist and have a levels-of-precaution framework and have a step where they actually stop and think before they begin.
(+1 from Rob B, Marie L)
_____________________________________________
Eliezer Yudkowsky: I worry that we’re shooting ourselves in the foot by telling ourselves that psychedelics “temporarily enable some self-modification capabilities” rather than doing shit to the brain that we don’t understand, and we know a bunch of people who seemed a lot more promising and sane before they did some psychedelics, and now they’re not the people they were anymore and not in a good way, and there is not in fact any good way to be sure of who that happens to because it did not seem very predictable in advance at the time, and maybe you can roll the dice on that if you’re tired of being yourself and want to take a bet with high variance and negative expected value, but you sure don’t do it in little subgroups that put an arm around somebody’s shoulder and make helpful offers.
(+1 from Rob B, Marie L, Jim B)
_____________________________________________
Jim Babcock: I suspect you may be underestimating the base rate of people using psychedelics discretely and having a neutral or mild-positive effect that they don’t talk about, and also underestimating the degree of stupidity that people bring to bear in their drug use.
At Burning Man, I saw a lot of stuff like: making dosing and mixing decisions while not sober; taking doses without a scale, not beig justifiably confident that the dose is the right order of magnitude; taking their epistemically-untrustworthy friend’s word that a substance they’ve never heard of before is safe for them. That sort of thing. And that’s before even getting into the social stuff, where eg people really shouldn’t be having conversations about crazymaking topics while high. And the legitimately hard stuff, like it’s important to have a tripsitter who will break thought loops if you step on a trauma trigger, but also important that that tripsitter not be a significant other or cult leader and not be someone who will talk to you about crazymaking topics.
Meanwhile nearly everyone has been exposed to extremely unsubtle and substantially false anti-drug propaganda, which fails to survive contact with reality. So it’s unfortunate but also unsurprising that the how-much-caution pendulum in their heads winds up swinging too far to the other side. The ideal messaging imo would leave most people feeling like planning an acid trip is more work than they personally will get around to, plus mild disdain towards impulsive usage and corner-cutting.
(+1 from Rob B, Marie L)
MIRI comments on Cotra’s “Case for Aligning Narrowly Superhuman Models”
AI Views Snapshots
MIRI location optimization (and related topics) discussion
Four mindset disagreements behind existential risk disagreements in ML
Rationality: From AI to Zombies
Yeah—I love AI_WAIFU’s comment, but I love the OP too.
To some extent I think these are just different strategies that will work better for different people; both have failure modes, and Eliezer is trying to guard against the failure modes of ‘Fuck That Noise’ (e.g., losing sight of reality), while AI_WAIFU is trying to guard against the failure modes of ‘Try To Die With More Dignity’ (e.g., losing motivation).
My general recommendation to people would be to try different framings / attitudes out and use the ones that empirically work for them personally, rather than trying to have the same lens as everyone else. I’m generally a skeptic of advice, because I think people vary a lot; so I endorse the meta-advice that you should be very picky about which advice you accept, and keep in mind that you’re the world’s leading expert on yourself. (Or at least, you’re in the best position to be that thing.)
Cf. ‘Detach the Grim-o-Meter’ versus ‘Try to Feel the Emotions that Match Reality’. Both are good advice in some contexts, for some people; but I think there’s some risk from taking either strategy too far, especially if you aren’t aware of the other strategy as a viable option.
Kate Donovan messaged me to say:
The base rate seems important to take into account here, though per Jessica, “Obviously, for every case of poor mental health that ‘blows up’ and is noted, there are many cases that aren’t.” (But I’d guess that’s true for the base-rate stats too?)