Thanks for the comment. I’m gonna err on the side of noting disagreements and giving brief descriptions of my perspective rather than writing something I think has a good chance of successfully persuading you of my perspective, primarily so as to actually write a reply in a timely fashion.
I don’t want to create an expectation that if you reply then you will reply to each point; rather I’d encourage you if you reply to simply reply to whichever points seem interesting or cruxy to you.
———
1) You make the analogy to having non-violent states. I concur that presently one cannot have states without militaries. I don’t see this as showing that in all domains one must maintain high offensive capabilities in order to have good defenses. I agree one needs defenses, but sometimes good defenses don’t look like “Training thousands of people how to carry out a targeted kill-strike” and instead look like “Not being tempted to reply by rude comments online” or “Checking whether a factual claim someone makes is accurate”.
You say that for LaSota and Brent that folks “could neither (a) adequately navigate their impact (myself included!) nor (b) rally ejection/exclusion power until well after they’d already had their impact” and “Maybe, you might hope, you can make the ejection/exclusion sensitivity refined enough to work earlier”.
I don’t share the sense of difficulty I read in the second of those quotes. I think the Bay Area rationalists (and most other rationalists globally) had some generally extreme lack of boundaries of any sort. The ~only legible boundaries that the Bay Area rationality scene had were (a) are you an employee at one of CFAR/MIRI, and (b) are you invited to CFAR events. MIRI didn’t have much to do with these two individuals, and I think CFAR was choosing a strategy of “we’re not really doing social policing, we’re primarily just selecting on people who have interesting ideas about rationality”. Everything else was highly social and friend-based and it was quite dramatic to ban people from your social events. The REACH was the only communityspace and if I recall correctly explicitly had no boundaries on who could be there. This is an environment where people with lots of red flags will be able to move around with much more ease than in the rest of the world.
I think these problems aren’t that hard once you have community spaces that are willing to enforce boundaries. Over the last few years I’ve run many events and spaces, and often gotten references for people who want to enter the spaces, and definitely chosen to not invite people due to concerns about ethics and responsible behavior. I don’t believe I would’ve accepted these two people into the spaces more than once or twice at most. It’s unpleasant work to enforce boundaries and I’ve made mistakes, but overall I think that there were just not many strong boundaries in these people’s way initially, and they would have been pushed back and dissuaded much earlier if there were.
2) You write:
But the most potent forms of distorted thinking aren’t about sorting out the logic. I think they look more like reaching deep down and finding ways to become immune to things like frame control.
My position is that most thinking isn’t really about reality and isn’t truth-tracking, but that if you are doing that thinking then a lot of important questions are surprisingly easy to answer. Generally doing a few fermi estimates with a few datapoints can get you pretty in touch with the relevant part of reality.
I think there’s a ton of adversarial stuff going on as well, but the primary reason that people haven’t noticed that AI is an x-risk isn’t because people are specifically trying to trick them about the domain, but because the people are not really asking themselves the question and checking.
(And also something about people not having any conception of what actions to take in the fact of a civilizational-scale problem that most of the rest of civilization is not thinking about.)
(I think there’s some argument to be made here that the primary reason people don’t think for themselves is because civilization is trying to make them go crazy, which is interesting, though I still think the solution is primarily “just make a space where you can actually think about the object level”.)
I acknowledge that there are people who are very manipulative and adversarial in illegible ways that are hard to pin down. There’s a whole discussion about offense/defense here and how it plays out. I currently expect that there are simple solutions here. As a pointer, someone I know and respect along with their partner, makes lists of people they know for whom they would not be surprised to later find out that the person did something quite manipulative/bad/unethical, and I think they’ve had some success with this. Also personally I have repeatedly kicked myself thinking “I knew that person was suspicious, why didn’t I say so earlier?” I don’t think these problems are particularly intractable and I do think people know things and I think probably there are good ways to help that info rise up and get shared (I do not claim to have solved this problem). I don’t think it requires you yourself being very skilled at engaging with manipulative people.
The standard rationalist defense I’ve noticed against this amounts to mental cramping. Demand everything go through cognition, and anything that seems to try to route around cognition gets a freakout/shutdown/”shame it into oblivion” kind of response.
Yeah I’ve seen this, and done it somewhat. I think it works in some situations, but there’s a bunch of adversarial situations it definitely doesn’t work. I do agree it seems like a false hope to think that this can be remotely sufficient.
3) I do sometimes look at people who think they’re at war a lot more than me, and they seem very paranoid and to spend so many cognitive cycles modeling ghosts and attacks that aren’t there. It seems so tiring! I suspect you and I disagree about the extent to which we are at war with people epistemically.
Another potentially relevant point here is that I tend to see large groups and institutions as the primary forces deceiving me and tricking me, and much less so individuals. I’m much more scared of Twitter winding its way into my OODA loop than I am of a selfish scheming frame controlling individual. I think it’s much easier for me to keep boundaries against individuals than I am against these much bigger and broader forces.
4)
All of which is to say:
I think a mature Art of Rationality would most definitely include something like skillful navigation of manipulation.
I don’t think every practitioner needs to master every aspect of a mature Art. Much like not all cooks need to know how to make a roux.
But an Art that has detection, exclusion, & avoidance as its only defense against Dark Artists is a much poorer & more vulnerable Art. IMO.
My perspective on these.
Personally I would like to know two or three people who have successfully navigated being manipulated, and hopefully have them write up their accounts of that.
I think aspiring rationalists should maneuver themselves into an environment where they can think clearly and be productive and live well, and maintain that, and not try to learn to survive being manipulated without a clear and present threat that they think they have active reason to move toward rather than away from.
I agree with your last claim. I note that when I read your comment I’m not sure whether you’re saying “this is an important area of improvement” or “this should be central to the art”, which are very different epistemic states.
I’m gonna err on the side of noting disagreements and giving brief descriptions of my perspective rather than writing something I think has a good chance of successfully persuading you of my perspective, primarily so as to actually write a reply in a timely fashion.
Acknowledged.
I don’t see this as showing that in all domains one must maintain high offensive capabilities in order to have good defenses.
Oh, uh, I didn’t mean to imply that. I meant to say that rejecting attention to military power is a bad strategy for defense. A much, much better defensive strategy is to study offense. But that doesn’t need to mean getting good at offense!
(Although I do think it means interacting with offense. Most martial arts fail spectacularly on this point for instance. Pragmatically speaking, you have to have practice actually defending yourself in order to get skillful at defense. And in cases like MMA, that does translate to getting skilled at attack! But that’s incidental. I think you could design good self-defense training systems that have most people never practicing offense.)
I think these problems aren’t that hard once you have community spaces that are willing to enforce boundaries. Over the last few years I’ve run many events and spaces, and often gotten references for people who want to enter the spaces, and definitely chosen to not invite people due to concerns about ethics and responsible behavior. I don’t believe I would’ve accepted these two people into the spaces more than once or twice at most.
Nice. And I agree, boundaries like this can be great for a large range of things.
I don’t think this helps the Art much though.
And it’s hard to know how much your approach doesn’t work.
I also wonder how much this lesson about boundaries arose because of the earlier Dark exploits. In which case it’s actually, ironically, an example of exactly the kind of thing I’m talking about! Only with lessons learned much more painfully than I think was necessary due to their not being sought out.
But also, maybe this is good enough for what you care about. Again, I don’t mean to pressure that you should do anything differently.
I’m mostly pushing back against the implication I read that “Nah, our patches are fine, we’ve got the Dark Arts distanced enough that they’re not an issue.” You literally can’t know that.
My position is that most thinking isn’t really about reality and isn’t truth-tracking, but that if you are doing that thinking then a lot of important questions are surprisingly easy to answer.
Totally agree. And this is a major defense against a lot of the stuff that bamboozles most folk.
I think there’s a ton of adversarial stuff going on as well, but the primary reason that people haven’t noticed that AI is an x-risk isn’t because people are specifically trying to trick them about the domain, but because the people are not really asking themselves the question and checking.
I agree — and I’m not sure why you felt this was relevant to say? I think maybe you thought I was saying something I wasn’t trying to.
(I think there’s some argument to be made here that the primary reason people don’t think for themselves is because civilization is trying to make them go crazy, which is interesting, though I still think the solution is primarily “just make a space where you can actually think about the object level”.)
This might be a crux between us. I’m not sure. But I think you might be seriously underestimating what’s involved in that “just” part (“just make a space…”). Attention on the object-level is key, I 100% agree there. But what defines the space? What protects its boundaries? If culture wants to grab you by the epistemic throat, but you don’t know how it tries to do so, and you just try to “make a space”… you’re going to end up way more confident of the clarity of your thinking than is true.
I acknowledge that there are people who are very manipulative and adversarial in illegible ways that are hard to pin down. […] …I think probably there are good ways to help that info rise up and get shared…. I don’t think it requires you yourself being very skilled at engaging with manipulative people.
I think there’s maybe something of a communication impasse happening here. I agree with what you’re saying here. I think it’s probably good enough for most cases you’re likely to care about, for some reasonable definition of “most”. It also strikes me as obvious that (a) it’s unlikely to cover all the cases you’re likely to care about, and (b) the Art would be deeply enriched by learning how one would skillfully engage with manipulative people. I don’t think everyone who wants to benefit from that enrichment needs to do that engagement, just like not everyone who wants to train in martial arts needs to get good at realistic self-defense.
I’ve said this several times, and you seem to keep objecting to my implied claim of not-that. I’m not sure what’s going on there. Maybe I’m missing your point?
I do sometimes look at people who think they’re at war a lot more than me, and they seem very paranoid and to spend so many cognitive cycles modeling ghosts and attacks that aren’t there. It seems so tiring!
I agree. I think it’s dumb.
I suspect you and I disagree about the extent to which we are at war with people epistemically.
Another potentially relevant point here is that I tend to see large groups and institutions as the primary forces deceiving me and tricking me, and much less so individuals.
Oh! I’m really glad you said this. I didn’t realize we were miscommunicating about this point.
I totally agree. This is what I mean when I’m talking about agents. I’m using adversarial individuals mostly as case studies & training data. The thing I actually care about is the multipolar war going on with already-present unaligned superintelligences. Those are the Dark forces I want to know how to be immune to.
I’m awfully suspicious of someone’s ability to navigate hostile psychofauna if literally their only defense against (say) a frame controller is “Sus, let’s exclude them.” You can’t exclude Google or wokism or collective anxiety the same way.
Having experienced frame control clawing at my face, and feeling myself become immune without having to brace… and noticing how that skill generalized to some of the tactics that the psychofauna use…
…it just seems super obvious to me that this is really core DADA. Non-cognitive, very deep, very key.
Personally I would like to know two or three people who have successfully navigated being manipulated, and hopefully have them write up their accounts of that.
Ditto!
I think aspiring rationalists should maneuver themselves into an environment where they can think clearly and be productive and live well, and maintain that, and not try to learn to survive being manipulated without a clear and present threat that they think they have active reason to move toward rather than away from.
Totally agree with the first part. I think the whole thing is a fine choice. I notice my stance of “Epistemic warriors would still be super useful” is totally unmoved thus far though. (And I’m reminded of your caveat at the very beginning!)
I’m reminded of the John Adams quote: “I must study Politicks and War that my sons may have liberty to study Mathematicks and Philosophy. My sons ought to study mathematics and Philosophy, Geography, natural History, naval Architecture, navigation, Commerce and Agriculature, in order to give their Children a right to study Painting, Poetry, Musick, Architecture, Statuary, Tapestry and Porcelaine.”
I note that when I read your comment I’m not sure whether you’re saying “this is an important area of improvement” or “this should be central to the art”, which are very different epistemic states.
Oh, I don’t know what should or shouldn’t be central to the Art.
It just strikes me that rationality currently is in a similar state as aikido.
Aikido claims to be an effective form of self-defense. (Or at least it used to! Maybe it’s been embarrassed out of saying that anymore?) It’s a fine practice, it has immense value… it’s just not what it says on the tin.
If it wanted to be what it claims, it needs to do things like add pressure testing. Realistic combat. Going into MMA tournaments and coming back with refinements to what it’s doing.
And that could be done in a way that honors its spirit! It can add the constraints that are key to its philosophy, like “Protect everyone involved, including the attacker.”
But maybe it doesn’t care about that. Maybe it just wants to be a sport and discipline.
That’s totally fine!
It does seem weird for it to continue claiming to be effective self-defense though. Like it needs its fake meaning to be something its practitioners believe in.
I think rationality is in a similar state. It has some really good stuff in it. Really good. It’s a great domain.
But I just don’t see it mattering for the power plays. I think rationalists don’t understand power, the same way aikido practitioners don’t understand fighting. And they seem to be in a similar epistemic state about it: they think they basically do, but they don’t pressure-test their understanding to check, best as I can tell.
So of your two options, it’s more like “important area for improvement”… roughly like pressure-testing could be an important area of improvement for aikido. It’d probably become a kind of central if it were integrated! But I don’t know.
And, I think the current state of rationality is fine.
Just weak in one axis it sometimes claims to care about.
COVID was one of the MMA-style arenas for different egregores to see which might come out ‘on top’ in an epistemically unfriendly environment.
I have a lot of opinions on this that are more controversial than I’m willing to go into right now. But I wonder what else will work as one of these “testing arenas.”
Thanks for the comment. I’m gonna err on the side of noting disagreements and giving brief descriptions of my perspective rather than writing something I think has a good chance of successfully persuading you of my perspective, primarily so as to actually write a reply in a timely fashion.
I don’t want to create an expectation that if you reply then you will reply to each point; rather I’d encourage you if you reply to simply reply to whichever points seem interesting or cruxy to you.
———
1) You make the analogy to having non-violent states. I concur that presently one cannot have states without militaries. I don’t see this as showing that in all domains one must maintain high offensive capabilities in order to have good defenses. I agree one needs defenses, but sometimes good defenses don’t look like “Training thousands of people how to carry out a targeted kill-strike” and instead look like “Not being tempted to reply by rude comments online” or “Checking whether a factual claim someone makes is accurate”.
You say that for LaSota and Brent that folks “could neither (a) adequately navigate their impact (myself included!) nor (b) rally ejection/exclusion power until well after they’d already had their impact” and “Maybe, you might hope, you can make the ejection/exclusion sensitivity refined enough to work earlier”.
I don’t share the sense of difficulty I read in the second of those quotes. I think the Bay Area rationalists (and most other rationalists globally) had some generally extreme lack of boundaries of any sort. The ~only legible boundaries that the Bay Area rationality scene had were (a) are you an employee at one of CFAR/MIRI, and (b) are you invited to CFAR events. MIRI didn’t have much to do with these two individuals, and I think CFAR was choosing a strategy of “we’re not really doing social policing, we’re primarily just selecting on people who have interesting ideas about rationality”. Everything else was highly social and friend-based and it was quite dramatic to ban people from your social events. The REACH was the only community space and if I recall correctly explicitly had no boundaries on who could be there. This is an environment where people with lots of red flags will be able to move around with much more ease than in the rest of the world.
I think these problems aren’t that hard once you have community spaces that are willing to enforce boundaries. Over the last few years I’ve run many events and spaces, and often gotten references for people who want to enter the spaces, and definitely chosen to not invite people due to concerns about ethics and responsible behavior. I don’t believe I would’ve accepted these two people into the spaces more than once or twice at most. It’s unpleasant work to enforce boundaries and I’ve made mistakes, but overall I think that there were just not many strong boundaries in these people’s way initially, and they would have been pushed back and dissuaded much earlier if there were.
2) You write:
My position is that most thinking isn’t really about reality and isn’t truth-tracking, but that if you are doing that thinking then a lot of important questions are surprisingly easy to answer. Generally doing a few fermi estimates with a few datapoints can get you pretty in touch with the relevant part of reality.
I think there’s a ton of adversarial stuff going on as well, but the primary reason that people haven’t noticed that AI is an x-risk isn’t because people are specifically trying to trick them about the domain, but because the people are not really asking themselves the question and checking.
(And also something about people not having any conception of what actions to take in the fact of a civilizational-scale problem that most of the rest of civilization is not thinking about.)
(I think there’s some argument to be made here that the primary reason people don’t think for themselves is because civilization is trying to make them go crazy, which is interesting, though I still think the solution is primarily “just make a space where you can actually think about the object level”.)
I acknowledge that there are people who are very manipulative and adversarial in illegible ways that are hard to pin down. There’s a whole discussion about offense/defense here and how it plays out. I currently expect that there are simple solutions here. As a pointer, someone I know and respect along with their partner, makes lists of people they know for whom they would not be surprised to later find out that the person did something quite manipulative/bad/unethical, and I think they’ve had some success with this. Also personally I have repeatedly kicked myself thinking “I knew that person was suspicious, why didn’t I say so earlier?” I don’t think these problems are particularly intractable and I do think people know things and I think probably there are good ways to help that info rise up and get shared (I do not claim to have solved this problem). I don’t think it requires you yourself being very skilled at engaging with manipulative people.
Yeah I’ve seen this, and done it somewhat. I think it works in some situations, but there’s a bunch of adversarial situations it definitely doesn’t work. I do agree it seems like a false hope to think that this can be remotely sufficient.
3) I do sometimes look at people who think they’re at war a lot more than me, and they seem very paranoid and to spend so many cognitive cycles modeling ghosts and attacks that aren’t there. It seems so tiring! I suspect you and I disagree about the extent to which we are at war with people epistemically.
Another potentially relevant point here is that I tend to see large groups and institutions as the primary forces deceiving me and tricking me, and much less so individuals. I’m much more scared of Twitter winding its way into my OODA loop than I am of a selfish scheming frame controlling individual. I think it’s much easier for me to keep boundaries against individuals than I am against these much bigger and broader forces.
4)
My perspective on these.
Personally I would like to know two or three people who have successfully navigated being manipulated, and hopefully have them write up their accounts of that.
I think aspiring rationalists should maneuver themselves into an environment where they can think clearly and be productive and live well, and maintain that, and not try to learn to survive being manipulated without a clear and present threat that they think they have active reason to move toward rather than away from.
I agree with your last claim. I note that when I read your comment I’m not sure whether you’re saying “this is an important area of improvement” or “this should be central to the art”, which are very different epistemic states.
Acknowledged.
Oh, uh, I didn’t mean to imply that. I meant to say that rejecting attention to military power is a bad strategy for defense. A much, much better defensive strategy is to study offense. But that doesn’t need to mean getting good at offense!
(Although I do think it means interacting with offense. Most martial arts fail spectacularly on this point for instance. Pragmatically speaking, you have to have practice actually defending yourself in order to get skillful at defense. And in cases like MMA, that does translate to getting skilled at attack! But that’s incidental. I think you could design good self-defense training systems that have most people never practicing offense.)
Nice. And I agree, boundaries like this can be great for a large range of things.
I don’t think this helps the Art much though.
And it’s hard to know how much your approach doesn’t work.
I also wonder how much this lesson about boundaries arose because of the earlier Dark exploits. In which case it’s actually, ironically, an example of exactly the kind of thing I’m talking about! Only with lessons learned much more painfully than I think was necessary due to their not being sought out.
But also, maybe this is good enough for what you care about. Again, I don’t mean to pressure that you should do anything differently.
I’m mostly pushing back against the implication I read that “Nah, our patches are fine, we’ve got the Dark Arts distanced enough that they’re not an issue.” You literally can’t know that.
Totally agree. And this is a major defense against a lot of the stuff that bamboozles most folk.
I agree — and I’m not sure why you felt this was relevant to say? I think maybe you thought I was saying something I wasn’t trying to.
This might be a crux between us. I’m not sure. But I think you might be seriously underestimating what’s involved in that “just” part (“just make a space…”). Attention on the object-level is key, I 100% agree there. But what defines the space? What protects its boundaries? If culture wants to grab you by the epistemic throat, but you don’t know how it tries to do so, and you just try to “make a space”… you’re going to end up way more confident of the clarity of your thinking than is true.
I think there’s maybe something of a communication impasse happening here. I agree with what you’re saying here. I think it’s probably good enough for most cases you’re likely to care about, for some reasonable definition of “most”. It also strikes me as obvious that (a) it’s unlikely to cover all the cases you’re likely to care about, and (b) the Art would be deeply enriched by learning how one would skillfully engage with manipulative people. I don’t think everyone who wants to benefit from that enrichment needs to do that engagement, just like not everyone who wants to train in martial arts needs to get good at realistic self-defense.
I’ve said this several times, and you seem to keep objecting to my implied claim of not-that. I’m not sure what’s going on there. Maybe I’m missing your point?
I agree. I think it’s dumb.
Oh! I’m really glad you said this. I didn’t realize we were miscommunicating about this point.
I totally agree. This is what I mean when I’m talking about agents. I’m using adversarial individuals mostly as case studies & training data. The thing I actually care about is the multipolar war going on with already-present unaligned superintelligences. Those are the Dark forces I want to know how to be immune to.
I’m awfully suspicious of someone’s ability to navigate hostile psychofauna if literally their only defense against (say) a frame controller is “Sus, let’s exclude them.” You can’t exclude Google or wokism or collective anxiety the same way.
Having experienced frame control clawing at my face, and feeling myself become immune without having to brace… and noticing how that skill generalized to some of the tactics that the psychofauna use…
…it just seems super obvious to me that this is really core DADA. Non-cognitive, very deep, very key.
Ditto!
Totally agree with the first part. I think the whole thing is a fine choice. I notice my stance of “Epistemic warriors would still be super useful” is totally unmoved thus far though. (And I’m reminded of your caveat at the very beginning!)
I’m reminded of the John Adams quote: “I must study Politicks and War that my sons may have liberty to study Mathematicks and Philosophy. My sons ought to study mathematics and Philosophy, Geography, natural History, naval Architecture, navigation, Commerce and Agriculature, in order to give their Children a right to study Painting, Poetry, Musick, Architecture, Statuary, Tapestry and Porcelaine.”
Oh, I don’t know what should or shouldn’t be central to the Art.
It just strikes me that rationality currently is in a similar state as aikido.
Aikido claims to be an effective form of self-defense. (Or at least it used to! Maybe it’s been embarrassed out of saying that anymore?) It’s a fine practice, it has immense value… it’s just not what it says on the tin.
If it wanted to be what it claims, it needs to do things like add pressure testing. Realistic combat. Going into MMA tournaments and coming back with refinements to what it’s doing.
And that could be done in a way that honors its spirit! It can add the constraints that are key to its philosophy, like “Protect everyone involved, including the attacker.”
But maybe it doesn’t care about that. Maybe it just wants to be a sport and discipline.
That’s totally fine!
It does seem weird for it to continue claiming to be effective self-defense though. Like it needs its fake meaning to be something its practitioners believe in.
I think rationality is in a similar state. It has some really good stuff in it. Really good. It’s a great domain.
But I just don’t see it mattering for the power plays. I think rationalists don’t understand power, the same way aikido practitioners don’t understand fighting. And they seem to be in a similar epistemic state about it: they think they basically do, but they don’t pressure-test their understanding to check, best as I can tell.
So of your two options, it’s more like “important area for improvement”… roughly like pressure-testing could be an important area of improvement for aikido. It’d probably become a kind of central if it were integrated! But I don’t know.
And, I think the current state of rationality is fine.
Just weak in one axis it sometimes claims to care about.
Musings:
COVID was one of the MMA-style arenas for different egregores to see which might come out ‘on top’ in an epistemically unfriendly environment.
I have a lot of opinions on this that are more controversial than I’m willing to go into right now. But I wonder what else will work as one of these “testing arenas.”