I read this post where you keep claiming you are banning him for ‘epistemic concerns’ but then link to 0 examples and mostly talk about some unrelated real-life thing which you also give 0 real explanation for.
The comments here mention a sex crime, but OP doesn’t. If that’s what happened why vaguebook, stay silent for a year and lie the ban’s for ‘epistemic concerns’? Who else have you banned for ‘epistemic concerns’ - nobody?
Honestly, after reading everything here I do have major concerns about ialdabaoth’s character, but the main epistemic concerns I have are about OP presenting this dishonestly after a year of silence.
My understanding is that the epistemic concern is “after writing the affordance widths post, he would tell young women he needed to do BDSM stuff they weren’t comfortable with in order to stay within his affordance width.” And similar things for some of his other posts. I’m not sure why the OP was so vague about this.
We believe that Brent is fundamentally oriented towards helping people grow to be the best versions of themselves. In this way he is aligned with CFAR’s goals and strategy and should be seen as an ally.
In particular, Brent is quite good at breaking out of standard social frames and making use of unconventional techniques and strategies. This includes things that have Chesterton’s fences attached, such as drug use, weird storytelling, etc. A lot of his aesthetic is dark, and this sometimes makes him come across as evil or machiavellian.
Brent also embodies a rare kind of agency and sense of heroic responsibility. This has caused him to take the lead in certain events and be an important community hub and driver. The flip side of this is that because Brent is deeply insecure, he has to constantly fight urges to seize power and protect himself. It often takes costly signalling for him to trust that someone is an ally, and even then it’s shaky.
Brent is a controversial figure, and disliked by many. This has led to him being attacked by many and held to a higher standard than most. In these ways his feelings of insecurity are justified. He also has had a hard life, including a traumatic childhood. Much of the reason people don’t like him comes from a kind of intuition or aesthetic feeling, rather than his actions per se.
“Everything bad that happened as a result of Brent’s actions is someone else’s fault. He only wants the best things to happen, but to allow that, everyone else needs to trust him first. Because bad people not trusting him is one of those things that make him do the things he regrets. Seriously, stop pointing out how Brent’s actions hurt people; don’t you see the pain this causes him?”
I knew a guy who did things similar to what Brent is accused of, and his PR was pretty similar. He also had a blog with complaints about his suffering and unfairness of the world. None of his written thoughts hinted at the possibility of him changing his own behavior; it was always how other people should accomodate him better.
My opinion on the articles is that they are a “fruit of the poisonous mind”. No matter how true or insightful, the text is optimized for some purpose. If you believe there is something valuable there, the only safe way is to throw the text away and try expressing the valuable idea using your own words, your own metaphors, and your own examples, as if the original text had never existed. Then you might notice e.g. that sometimes the space of action is not one-dimensional, however people are often blind about some options they have.
jimrandomh’s comment, linked in the OP, is the current best explanation of the epistemic concerns. Most of my impression comes from in-person discussions with ialdabaoth, which I obviously can’t link to, and Facebook posts, of which relatively few individual posts would be damning in isolation but the whole of them adds up to a particular impression. I also want to stress that this is not just my opinion; I currently know of no person who would describe ialdabaoth as a good influence, merely people who think he is redeemable. [I’d be happy to edit this if anyone wants to defend him privately; I have tried to counteract the effect where people might be unwilling to defend him publicly because of the unpopularity of defending a man accused of sexual assault.]
I currently don’t have an interest in writing a post with this level of detail explaining what I’m concerned about, because it has massive costs and unclear benefits. One of ialdabaoth’s virtues is that he was cooperative about leaving; rather than us having to figure out whether or not we wanted to ban him, he stopped commenting and posting to not force the issue, which is why this is happening now instead of before. If I were going to invest that level of effort into something, it would be people who I think are actively making things worse now.
jimrandomh’s comment, linked in the OP, is the current best explanation of the epistemic concerns.
Excluding the personal stuff, this comment is just a somewhat standard LW critique of a LW post (which has less karma than the original post fwiw). If this is the criteria for an ‘epistemic concerns’ ban, then you must’ve banned hundreds of people. If you haven’t you are clearly banning him for the other reasons, I don’t know why you insist on being dishonest about it.
He is using this comment to show the ‘epistemic concerns’ side specifically, and claiming the personal stuff were separate .
This is the specific claim.
We think that ialdabaoth poses a substantial risk to our epistemic environment due to manipulative epistemic tactics, based on our knowledge and experience of him. This is sufficient reason for the ban, and holds without investigating or making any sort of ruling on other allegations.
Maybe I’m confused about what you mean by “the personal stuff”. My impression is that what I would consider “the personal stuff” is central to why ialdabaoth is considered to pose an epistemic threat: he has (allegedly) a history of manipulation which makes it more likely that any given thing he writes is intended to deceive or manipulate. Which is why jimrandomh said:
The problem is, I think this post may contain a subtle trap, and that understanding its author, and what he was trying to do with this post, might actually be key to understanding what the trap is.
and, by way of explanation of why “this post may contain a subtle trap”, a paragraph including this:
So he created narratives to explain why those conversations were so confusing, why he wouldn’t follow the advice, and why the people trying to help him were actually wronging him, and therefore indebted. This post is one such narrative.
Unless I’m confused, (1) this is not “a somewhat standard LW critique of a LW post” because most such critiques don’t allege that the thing critiqued is likely to contain subtle malignly-motivated traps, and (2) the reason for taking it seriously is “the personal stuff”.
Who’s saying, in what sense, that “the personal stuff were separate”?
Vaniver is saying that the personal stuff didn’t come into account when banning him and that epistemic concerns were enough. From OP:
We think that ialdabaoth poses a substantial risk to our epistemic environment due to manipulative epistemic tactics, based on our knowledge and experience of him. This is sufficient reason for the ban, and holds without investigating or making any sort of ruling on other allegations.
but then the epistemic concerns seem to be purely based on stuff from the “other allegations” part.
And honestly, the quality of that post is (subjectively) higher than the quality of > 99% of current LW posts, yet the claim is that content is what he is banned for, which is a bit ridiculous. What I am asking is, why pretend it is the content and that the “other allegations” have no part?
What I am asking is, why pretend it is the content and that the “other allegations” have no part?
As mentioned in a sibling comment, I am trying to establish the principle that ‘promoting reasoning styles in a way we think is predatory’ can be a bannable offense, independent of whether or not predation has obviously happened, in part because I think that’s part of having a well-kept garden and in part so that the next person in ialdabaoth’s reference class can be prevented from doing significant harm. Simply waiting until someone has been exiled doesn’t do that.
I don’t think I have the slightest idea what this means, even having read the OP and everything else about ialdabaoth’s actions. That’s a problem.
I am not worried about you in this regard; if anyone is interested in whether or not they should be worried for themselves, please reach out to me.
Many sorts of misbehavior can be adequately counteracted by clear rules; a speed limit is nicely quantitative and clearly communicable. Misbehavior on a higher level of abstraction like “following the letter of the law but not the spirit” cannot be adequately counteracted in the same sort of way. If one could letter out the spirit of the law, they would have done that the first time. Similarly, if I publish my sense of “this is how I detect adversarial reasoners,” then an adversarial reasoner has an easier time getting past my defenses.
I will put some thought into whether I can come up with a good discussion of what I mean by ‘predatory’, assuming that’s where the confusion is; if instead it’s in something like “promoting reasoning styles” I’d be happy to attempt to elaborate that.
If one could letter out the spirit of the law, they would have done that the first time.
Some sort of typo / word substitution here, I think? Well, I think I get the sense of what you meant here from context. Anyhow…
Similarly, if I publish my sense of “this is how I detect adversarial reasoners,” then an adversarial reasoner has an easier time getting past my defenses.
I appreciate that, but this was not the point of my comment. Rather, I was saying that it’s entirely unclear to me what you are even trying to detect. (“Adversarial reasoners”, yes, but that is obviously far too broad a category… isn’t it? Maybe I don’t know what you mean by that, either…)
I will put some thought into whether I can come up with a good discussion of what I mean by ‘predatory’, assuming that’s where the confusion is; if instead it’s in something like “promoting reasoning styles” I’d be happy to attempt to elaborate that.
Both, in fact. I would appreciate some elaboration on both counts!
Some sort of typo / word substitution here, I think? Well, I think I get the sense of what you meant here from context. Anyhow…
Yeah, I tried to shorten “If one could capture the spirit of the law in letters, they would have done that the first time.” and I don’t think it worked very well.
Both, in fact. I would appreciate some elaboration on both counts!
So thinking about this response to Zack_M_Davis made me realize that there’s a bit of my model that I might be able to explain easily.
I often think of things like LessWrong as having a ‘gravity well’, where people come close and then get drawn in by some features that they want more of. Gravity is obviously disanalogous in several ways, as it ignores the role played by preferences (on seeing something, some people become more interested whereas others become more bored), but I think that doesn’t affect the point too much; the main features that I want are something like “people near the center feel a stronger pull to stay in than people further away, and the natural dynamics tend to pull people closer over time.” I think often a person is in many such wells at once, and there’s some dynamic equilibrium between how all their different interests want to pull them.
Sometimes, people want to have a gravity well around themselves to pull other people closer to them. This class contains lots of ordinary things; someone starting a company and looking to build a team does this, someone trying to find apprentices or otherwise mentor people does this, someone attempting to find a romantic partner does a narrower version of this.
Whether this is ‘predatory’ seems to be a spectrum that for me mostly depends on a few factors, primarily how relative benefit is being prioritized and how information asymmetries are being handled. A clearly non-predatory case is Gallant attempting to start a company where it will succeed and all founders / early employees / investors will share in the profits and whose customers will be satisfied, and a clearly predatory case is Goofus attempting to start a company where the main hope is to exit before things collapse, and plans to pay very little by promising lots of equity while maintaining sufficient control that all of the early employees can be diluted out of profits when the time comes.
One way to have a gravity well around you is to create something new and excellent that people flock to. Another way to have a gravity well pulling other people towards you is moving towards the center of a pre-existing well. Imagine someone getting good at a martial art so that they can teach that martial art to others and feel knowledgeable and helpful.
A third option is becoming a ‘fringe master’, where you create something new at the boundary of something pre-existing. You don’t have to pay the costs of being part of the ‘normal system’ and dealing with its oversight, but do gain many of the benefits of its advertising / the draw of the excellence at the center of the pre-existing thing. This is basically the category that I am most worried about / want to be able to act against, where someone will take advantage of new people drawn in by LessWrong / the rationality community who don’t know about the missing stairs to watch out for or who is held in low regard. A general feature of this case is that the most unsavory bits will happen in private, or only be known through rumor, or assessments of technical skill or correctness that seem obvious to high-skill individuals will not be widely held.
Put another way, it seems to me like the rationality community is trying to draw people in, so that they can get the benefits of rationality / they can improve rationality / they can contribute to shared projects, and it would be good to devote attention to the edges of the community and making sure that we’re not losing marginal people we attract to traps set up on the edge of the community.
Furthermore, given the nature of our community, I expect those traps to look a lot like reasoning styles or models that contain traps that the people absorbing the model are unable to see. One of the main things that rationalists do is propose ways of thinking things through that could lead to better truth-tracking or superior performance in some way.
The primary thing I’m worried about there is when the superior performance is on a metric like “loyalty to the cause” or “not causing problems for ialdabaoth.” On a top-level comment, Isnasene writes:
Using Affordance Widths to manipulate people into doing things for you is basically a fancy pseudo-rationalist way of manipulating people into doing things for you by making them feel guilty and responsible.
Which is basically my sense; the general pattern was that ialdabaoth would insist on his frame / interpretation that he was being oppressed by society / the other people in the conversation, and would use a style of negotiation that put the pressure on the other person to figure out how to accommodate him / not be guilty of oppressing him instead of on him to not violate their boundaries. There’s also a general sense that he was aware the of the risk to him for open communication or clear thinking about about him, and thus would work against both when possible.
becoming a ‘fringe master’, where you create something new at the boundary of something pre-existing. You don’t have to pay the costs of being part of the ‘normal system’ and dealing with its oversight, but do gain many of the benefits of its advertising / the draw of the excellence at the center of the pre-existing thing. This is basically the category that I am most worried about / want to be able to act against, where someone will take advantage of new people drawn in by LessWrong / the rationality community who don’t know about the missing stairs to watch out for or who is held in low regard.
Thanks for explaining this part; this is really helpful. This model seems to assume that the “oversight” of the “normal system” at the center of the gravity well is trustworthy. I’m currently most worried[1] about the scenario where the “normal system” is corrupt: that new people are getting drawn in by the awesomeness of the Sequences and Harry Potter and the Methods,[2] only to get socialized into a community whose leadership and dominant social trend pays lip service to “rationality”, but is not actually interested in using reasoning (at least, not in public) when using reasoning would be socially inconvenient (whether due to the local area’s political environment, the asymmetric incentives faced by all sufficiently large organizations, the temptation to wirehead on our own “rationality” and “effectiveness” marketing promises, or many other possible reasons) and therefore require a small amount of bravery.
The worst thing for an epistemic standard is not the person who ignores or denies it, but the person who tries to mostly follow it when doing so feels right or is convenient while not acknowledging that they aren’t following it when it feels weird or inconvenient, as that leads to a community of people with such standards engaging in double-think WRT whether their standards call for weird or inconvenient behavior.
Have you thought at all about how to prevent the center of the gravity well from becoming predatory? Obviously, I’m all in favor of having systems to catch missing-stair rapists. But if you’re going to build an “immune system” to delegitimize anyone “held in low regard” without having to do the work of engaging with their arguments—without explaining why an amorphous mob that holds something in “low regard” can be trusted to reach that judgement for reliably good reasons—then you’re just running a cult. And if enough people who remember the spirit of the Sequences notice their beloved rationality community getting transformed into a cult, then you might have a rationalist civil war on your hands.
(Um, sorry if that’s too ominous or threatening of a phrasing. I think we mostly want the same thing, but have been following different strategies and exposed to different information, and I notice myself facing an incentive to turn up the rhetoric and point menacingly at my BATNA in case that helps with actually being listened to, because recent experiences have trained my brain to anticipate that even high-ranking “rationalists” are more interested in avoiding social threat than listening to arguments. As I’m sure you can also see, this is already a very bad sign of the mess we’re in.)
“Worried” is an understatement. It’s more like panicking continuously all year with many hours of lost sleep, crying fits, pacing aimlessly instead of doing my dayjob, and eventually doing enough trauma processing to finish writing my forthcoming 20,000-word memoir explaining in detail (as gently and objectively as possible while still telling the truth about my own sentiments and the world I see) why you motherfuckers are being incredibly intellectually dishonest (with respect to a sense of “intellectual dishonesty” that’s about behavior relative to knowledge, not conscious verbal “intent”).
Notably, written at a time Yudkowsky and “the community” had a lower public profile and therefore faced less external social pressure. This is not a coincidence because nothing is ever a coincidence.
This model seems to assume that the “oversight” of the “normal system” at the center of the gravity well is trustworthy.
On the core point, I think you improve / fix problems with the normal system in the boring, hard ways, and do deeply appreciate you championing particular virtues even when I disagree on where the balance of virtues lies.
I find something offputting here about the word “trustworthy,” because I feel like it’s a 2-place word; I think of oversight as something like “good enough to achieve standard X”, whereas “trustworthy” alone seems to imply there’s a binary standard that is met or not (and has been met). It seems like we could easily have very different standards for trustworthiness that cause us to not disagree on the facts while disagreeing on the implications.
(Somehow, it reminds me of this post and Caledonian’s reaction to it.)
Have you thought at all about how to prevent the center of the gravity well from becoming predatory?
Yes. Mostly this has focused on recruitment work for MIRI, where we really don’t want to guilt people into working on x-risk reduction (as not only is it predatory, it also is a recipe for them burning out instead of being productive, and so morality and efficiency obviously align), and yet most of the naive ways to ask people to consider working on x-risk reduction risk guilting them, and you need a more sophisticated way to remove that failure mode than just saying “please don’t interpret this as me guilting you into it!”. This is a thing that I’ve already written that’s parts of my longer thoughts here.
And, obviously, when I think about moderating LessWrong, I think about how to not become corrupt myself, and what sorts of habits and systems lower the chances of that, or make it more obvious if it does happen.
it’s a 2-place word [...] It seems like we could easily have very different standards for trustworthiness that cause us to not disagree on the facts while disagreeing on the implications.
Right, I agree that we don’t want to get into a pointless pseudo-argument where everyone agrees that x = 60, and yet we have a huge shouting match over whether this should be described using the English word “large” or “small.”
Maybe a question that would lead to a more meaningful disagreement would be, “Should our culture become more or less centralized?”—where centralized is the word I’m choosing to refer to a concept I’m going to try to describe extensionally/ostensively in the following two paragraphs.[1]
In a high-centralization culture, there’s a stronger presumption that our leaders in the town center come closer to knowing everything already, and that the reasoning styles or models being hawked by fringe masters are likely to “contain traps that the people absorbing the model are unable to see”: that is, thinking for yourself doesn’t work. As a result, our leaders might talk up “the value of having a community-wide immune system” so that they can “act against people who are highly manipulative and deceitful before they have clear victims.” If a particular fringe master starts becoming popular, our leaders might want to announce that they are “actively hostile to [the fringe master], and make it clear that [we] do not welcome support from those quarters.”
You seem to be arguing that we should become more centralized. I think that would be moving our culture in the absolute wrong direction. As long as we’re talking about patterns of adversarial optimization, I have to say that, to me, this kind of move looks optimized for “making it easier to ostracize and silence people who could cause trouble for MIRI and CfAR (e.g., Vassar or Ziz), either by being persistent critics or by embarrassing us in front of powerful third parties who are using guilt-by-association heuristics”, rather than improving our collective epistemics.
This seems like a substantial disagreement, rather than a trivial Sorites problem about how to use the word “trustworthy”.
do deeply appreciate you championing particular virtues even when I disagree on where the balance of virtues lies
not because most fringe masters are particularly good (they aren’t), but because thinking for yourself actually works and it’s not like our leaders in the town center know everything already.
I think the leaders in the town center do not know everything already. I think different areas have different risks when it comes to “thinking for yourself.” It’s one thing to think you can fly and jump off a roof yourself, and another thing to think it’s fine to cook for people when you’re Typhoid Mary, and I worry that you aren’t drawing a distinction here between those cases.
I have thought about this a fair amount, but am not sure I’ve discovered right conceptual lines here, and would be interested in how you would distinguish between the two cases, or if you think they are fundamentally equivalent, or that one of them isn’t real.
You seem to be arguing that we should become more centralized. I think that would be moving our culture in the absolute wrong direction.
In short, I think there are some centralizing moves that are worth it, and others that aren’t, and that we can choose policies individually instead of just throwing the lever on “centralization: Y/N”. Well-Kept Gardens Die by Pacifism is ever relevant; here, the thing that seems relevant to me is that there are some basic functions that need to happen (like, say, the removal of spam), and fulfilling those functions requires tools that could also be used for nefarious functions (as we could just mark criticisms of MIRI as ‘spam’ and they would vanish). But the conceptual categories that people normally have for this are predicated on the interesting cases; sure, both Nazi Germany and WWII America imprisoned rapists, but the interesting imprisonments are of political dissidents, and we might prefer WWII America because it had many fewer such political prisoners, and further prefer a hypothetical America that had no political prisoners. But this spills over into the question of whether we should have prisons or justice systems at all, and I think people’s intuitions on political dissidents are not very useful for what should happen with the more common sort of criminal.
Like, it feels almost silly to have to say this, but I like it when people put forth public positions that are critical of an idea I favor, because then we can argue about it and it’s an opportunity for me to learn something, and I generally expect the audience to be able to follow it and get things right. Like, I disagreed pretty vociferously with The AI Timelines Scam, and yet I thought the discussion it prompted was basically good. It did not ping my Out To Get You sensors in the way that ialdabaoth does. To me, this feels like a central example of the sort of thing you see in a less centralized culture where people are trying to think things through for themselves and end up with different answers, and is not at risk from this sort of moderation.
“Worried” is an understatement. It’s more like panicking continuously all year with many hours of lost sleep, crying fits, pacing aimlessly instead of doing my dayjob, and eventually doing enough trauma processing to finish writing my forthcoming 20,000-word memoir explaining in detail (as gently and objectively as possible while still telling the truth about my own sentiments and the world I see) why you motherfuckers are being incredibly intellectually dishonest (with respect to a sense of “intellectual dishonesty” that’s about behavior relative to knowledge, not conscious verbal “intent”).
I think I’m someone who might be sympathetic to your case, but just don’t understand what it is, so I’m really curious about this “memoir”. (Let me know if you want me to read your current draft.) Is this guess, i.e., you’re worried about the community falling prey to runaway virtue signaling, remotely close?
The separation I’m hoping to make is between banning him because “we know he committed sex crimes” and banning him because “he’s promoting reasoning styles in a way we think is predatory.” We do not know the first to the standard of legal evidence; ialdabaoth has not been convicted in a court of law, and while I think the community investigations were adequate for the questions of whether or not he should be allowed at particular events or clubs, my sense is that his exile was decided by amorphous community-wide processing in a way that I’m reluctant to extend further.
I’m making the additional claim that “The Bay Area community has a rough consensus to kick this guy out” by itself does not meet my bar for banning someone from LessWrong, given the different dynamics of in-person and online interactions. (As a trivial example, suppose someone’s body has a persistent and horrible smell; it could easily be the utilitarian move to not allow them at any physical meetups while giving them full freedom to participate online.) I think this is the bit Tenoke is finding hardest to swallow; it’s one thing to say “yep, this guy is exiled and we’re following the herd” and another thing to say “we’ve exercised independent judgment here, despite obvious pressures to conform.” The latter is a more surprising claim, and correspondingly would require more evidence.
and (2) the reason for taking it seriously is “the personal stuff”.
I think this is indirectly true. That is, there’s a separation between expected harm and actual harm, and I’m trying to implement procedures that reduce expected harm. Consider the difference between punishing people for driving drunk and just punishing people for crashing. It’s one thing to just wait until someone accumulates a cloud of ‘unfortunate events’ around them that leads to them finally losing their last defenders, and another to take active steps to assess risks and reduce them. Note that this requires a good model of how ‘drunkenness’ leads to ‘crashes’, and I do not see us as having presented a convincing model of that in this case.
Of course, this post isn’t an example of that; as mentioned, this post is years late, and the real test of whether we can do the equivalent of punishing people for driving drunk is whether we can do anything about people currently causing problems [in expectation]. But my hope is that this community slowly moves from a world where ‘concerns about X’ are published years after they’ve become mutual knowledge among people in the know to one where corrosive forces are actively cleaned up before they make things substantially worse.
I currently know of no person who would describe ialdabaoth as a good influence
“Good influence” in what context? I remember finding some of his Facebook posts/comments insightful, and I’m glad I read them. It shouldn’t be surprising that someone could have some real insights (and thereby constitute a “good influence” in the capacity of social-studies blogging), while also doing a lot of bad things (and thereby constituting a very bad influence in the capacity of being a real-life community member), even if the content of the insights is obviously related to the bad things. (Doing bad things and getting away with them for a long time requires skills that might also lend themselves to good social-studies blogging.)
Less Wrong is in the awkward position of being a public website (anyone can submit blog posts about rationality under a made-up name), and also being closely associated with a real-life community with dense social ties, group houses, money, &c. If our actual moderation algorithm is, “Ban people who have have been justly ostracized from the real-life community as part of their punishment, even if their blog comments were otherwise OK”, that’s fine, but we shouldn’t delude ourselves about what the algorithm is.
I remember finding some of his Facebook posts/comments insightful, and I’m glad I read them.
If you learned that someone in the rationality community had taken on ialdabaoth as a master (like in the context of zen, or a PhD advisor, or so on), would you expect them to grow in good directions or bad directions? [Ideally, from the epistemic state you were in ~2 years ago, rather than the epistemic state you’re in now.]
I acknowledge that this is quite different from the “would you ever appreciate coming across their thoughts?” question; as it happens, I upvoted Affordance Widths when I first saw it, because it was a neat simple presentation of a model of privilege, and wasn’t taking responsibility for his collected works or thinking about how it fit into them. I mistakenly typical-minded on which parts of his work people were listening to, and which they were safely ignoring.
If our actual moderation algorithm is, “Ban people who have have been justly ostracized from the real-life community as part of their punishment, even if their blog comments were otherwise OK”, that’s fine, but we shouldn’t delude ourselves about what the algorithm is.
I agree that algorithm could be fine, for versions of LessWrong that are focused primarily on the community instead of on epistemic progress, but am trying to help instantiate the version of LessWrong that is primarily focused on epistemic progress.
If you learned that someone in the rationality community had taken on ialdabaoth as a master (like in the context of zen, or a PhD advisor, or so on), would you expect them to grow in good directions or bad directions?
Bad directions. The problem is that I also think this of other users who we aren’t banning, which suggests that our standard for “allowed to post on Less Wrong” is lower than “would be a good master.”
[Ideally, from the epistemic state you were in ~2 years ago, rather than the epistemic state you’re in now.]
Okay, right, I’m much less sure that I would have said “bad directions” as confidently 2 years ago.
am trying to help instantiate the version of LessWrong that is primarily focused on epistemic progress.
Thanks for this!! (I think I’m much more worried than you about the failure mode where something claiming to make intellectual progress is actually doing something else, which makes me more willing to tolerate pragamatic concessions of principle that are explicitly marked as such, when I’m worried that the alternative is the concession being made anyway with a fake rationale attached.)
The problem is that I also think this of other users who we aren’t banning, which suggests that our standard for “allowed to post on Less Wrong” is lower than “would be a good master.”
I interpreted ialdabaoth as trying to be a master, in a way that I do not interpret most of the people who fail my “would be a good master” check. (Most of them are just not skilled in that sort of thing and don’t seek it out.) If there are people you think would both predictably mislead their students and appear to be trying to recruit students, I’m interested in hearing about it.
I think I’m much more worried than you about the failure mode where something claiming to make intellectual progress is actually doing something else
This is possible; my suspicion is that we have similar levels of dispreference for that and different models of how attempts to make intellectual progress go astray. I try to be upfront about when I make pragmatic concessions, as I think you’ve have some evidence of, in part so that when I am not making such a concession it’s more believable. [Of course, for someone without observations of that sort of thing, I don’t expect me claiming that I do it to be much evidence.]
I read this post where you keep claiming you are banning him for ‘epistemic concerns’ but then link to 0 examples and mostly talk about some unrelated real-life thing which you also give 0 real explanation for.
The comments here mention a sex crime, but OP doesn’t. If that’s what happened why vaguebook, stay silent for a year and lie the ban’s for ‘epistemic concerns’? Who else have you banned for ‘epistemic concerns’ - nobody?
Honestly, after reading everything here I do have major concerns about ialdabaoth’s character, but the main epistemic concerns I have are about OP presenting this dishonestly after a year of silence.
My understanding is that the epistemic concern is “after writing the affordance widths post, he would tell young women he needed to do BDSM stuff they weren’t comfortable with in order to stay within his affordance width.” And similar things for some of his other posts. I’m not sure why the OP was so vague about this.
Edit: and he also managed to fool members of CFAR with a similar line.
Reading the ACDC conclusion, it feels like:
“Everything bad that happened as a result of Brent’s actions is someone else’s fault. He only wants the best things to happen, but to allow that, everyone else needs to trust him first. Because bad people not trusting him is one of those things that make him do the things he regrets. Seriously, stop pointing out how Brent’s actions hurt people; don’t you see the pain this causes him?”
I knew a guy who did things similar to what Brent is accused of, and his PR was pretty similar. He also had a blog with complaints about his suffering and unfairness of the world. None of his written thoughts hinted at the possibility of him changing his own behavior; it was always how other people should accomodate him better.
My opinion on the articles is that they are a “fruit of the poisonous mind”. No matter how true or insightful, the text is optimized for some purpose. If you believe there is something valuable there, the only safe way is to throw the text away and try expressing the valuable idea using your own words, your own metaphors, and your own examples, as if the original text had never existed. Then you might notice e.g. that sometimes the space of action is not one-dimensional, however people are often blind about some options they have.
jimrandomh’s comment, linked in the OP, is the current best explanation of the epistemic concerns. Most of my impression comes from in-person discussions with ialdabaoth, which I obviously can’t link to, and Facebook posts, of which relatively few individual posts would be damning in isolation but the whole of them adds up to a particular impression. I also want to stress that this is not just my opinion; I currently know of no person who would describe ialdabaoth as a good influence, merely people who think he is redeemable. [I’d be happy to edit this if anyone wants to defend him privately; I have tried to counteract the effect where people might be unwilling to defend him publicly because of the unpopularity of defending a man accused of sexual assault.]
I currently don’t have an interest in writing a post with this level of detail explaining what I’m concerned about, because it has massive costs and unclear benefits. One of ialdabaoth’s virtues is that he was cooperative about leaving; rather than us having to figure out whether or not we wanted to ban him, he stopped commenting and posting to not force the issue, which is why this is happening now instead of before. If I were going to invest that level of effort into something, it would be people who I think are actively making things worse now.
Excluding the personal stuff, this comment is just a somewhat standard LW critique of a LW post (which has less karma than the original post fwiw). If this is the criteria for an ‘epistemic concerns’ ban, then you must’ve banned hundreds of people. If you haven’t you are clearly banning him for the other reasons, I don’t know why you insist on being dishonest about it.
Why would it make sense to “exclude the personal stuff”? Isn’t the personal stuff the point here?
He is using this comment to show the ‘epistemic concerns’ side specifically, and claiming the personal stuff were separate .
This is the specific claim.
Maybe I’m confused about what you mean by “the personal stuff”. My impression is that what I would consider “the personal stuff” is central to why ialdabaoth is considered to pose an epistemic threat: he has (allegedly) a history of manipulation which makes it more likely that any given thing he writes is intended to deceive or manipulate. Which is why jimrandomh said:
and, by way of explanation of why “this post may contain a subtle trap”, a paragraph including this:
Unless I’m confused, (1) this is not “a somewhat standard LW critique of a LW post” because most such critiques don’t allege that the thing critiqued is likely to contain subtle malignly-motivated traps, and (2) the reason for taking it seriously is “the personal stuff”.
Who’s saying, in what sense, that “the personal stuff were separate”?
Vaniver is saying that the personal stuff didn’t come into account when banning him and that epistemic concerns were enough. From OP:
but then the epistemic concerns seem to be purely based on stuff from the “other allegations” part.
And honestly, the quality of that post is (subjectively) higher than the quality of > 99% of current LW posts, yet the claim is that content is what he is banned for, which is a bit ridiculous. What I am asking is, why pretend it is the content and that the “other allegations” have no part?
As mentioned in a sibling comment, I am trying to establish the principle that ‘promoting reasoning styles in a way we think is predatory’ can be a bannable offense, independent of whether or not predation has obviously happened, in part because I think that’s part of having a well-kept garden and in part so that the next person in ialdabaoth’s reference class can be prevented from doing significant harm. Simply waiting until someone has been exiled doesn’t do that.
I don’t think I have the slightest idea what this means, even having read the OP and everything else about ialdabaoth’s actions. That’s a problem.
I am not worried about you in this regard; if anyone is interested in whether or not they should be worried for themselves, please reach out to me.
Many sorts of misbehavior can be adequately counteracted by clear rules; a speed limit is nicely quantitative and clearly communicable. Misbehavior on a higher level of abstraction like “following the letter of the law but not the spirit” cannot be adequately counteracted in the same sort of way. If one could letter out the spirit of the law, they would have done that the first time. Similarly, if I publish my sense of “this is how I detect adversarial reasoners,” then an adversarial reasoner has an easier time getting past my defenses.
I will put some thought into whether I can come up with a good discussion of what I mean by ‘predatory’, assuming that’s where the confusion is; if instead it’s in something like “promoting reasoning styles” I’d be happy to attempt to elaborate that.
Some sort of typo / word substitution here, I think? Well, I think I get the sense of what you meant here from context. Anyhow…
I appreciate that, but this was not the point of my comment. Rather, I was saying that it’s entirely unclear to me what you are even trying to detect. (“Adversarial reasoners”, yes, but that is obviously far too broad a category… isn’t it? Maybe I don’t know what you mean by that, either…)
Both, in fact. I would appreciate some elaboration on both counts!
Yeah, I tried to shorten “If one could capture the spirit of the law in letters, they would have done that the first time.” and I don’t think it worked very well.
So thinking about this response to Zack_M_Davis made me realize that there’s a bit of my model that I might be able to explain easily.
I often think of things like LessWrong as having a ‘gravity well’, where people come close and then get drawn in by some features that they want more of. Gravity is obviously disanalogous in several ways, as it ignores the role played by preferences (on seeing something, some people become more interested whereas others become more bored), but I think that doesn’t affect the point too much; the main features that I want are something like “people near the center feel a stronger pull to stay in than people further away, and the natural dynamics tend to pull people closer over time.” I think often a person is in many such wells at once, and there’s some dynamic equilibrium between how all their different interests want to pull them.
Sometimes, people want to have a gravity well around themselves to pull other people closer to them. This class contains lots of ordinary things; someone starting a company and looking to build a team does this, someone trying to find apprentices or otherwise mentor people does this, someone attempting to find a romantic partner does a narrower version of this.
Whether this is ‘predatory’ seems to be a spectrum that for me mostly depends on a few factors, primarily how relative benefit is being prioritized and how information asymmetries are being handled. A clearly non-predatory case is Gallant attempting to start a company where it will succeed and all founders / early employees / investors will share in the profits and whose customers will be satisfied, and a clearly predatory case is Goofus attempting to start a company where the main hope is to exit before things collapse, and plans to pay very little by promising lots of equity while maintaining sufficient control that all of the early employees can be diluted out of profits when the time comes.
One way to have a gravity well around you is to create something new and excellent that people flock to. Another way to have a gravity well pulling other people towards you is moving towards the center of a pre-existing well. Imagine someone getting good at a martial art so that they can teach that martial art to others and feel knowledgeable and helpful.
A third option is becoming a ‘fringe master’, where you create something new at the boundary of something pre-existing. You don’t have to pay the costs of being part of the ‘normal system’ and dealing with its oversight, but do gain many of the benefits of its advertising / the draw of the excellence at the center of the pre-existing thing. This is basically the category that I am most worried about / want to be able to act against, where someone will take advantage of new people drawn in by LessWrong / the rationality community who don’t know about the missing stairs to watch out for or who is held in low regard. A general feature of this case is that the most unsavory bits will happen in private, or only be known through rumor, or assessments of technical skill or correctness that seem obvious to high-skill individuals will not be widely held.
Put another way, it seems to me like the rationality community is trying to draw people in, so that they can get the benefits of rationality / they can improve rationality / they can contribute to shared projects, and it would be good to devote attention to the edges of the community and making sure that we’re not losing marginal people we attract to traps set up on the edge of the community.
Furthermore, given the nature of our community, I expect those traps to look a lot like reasoning styles or models that contain traps that the people absorbing the model are unable to see. One of the main things that rationalists do is propose ways of thinking things through that could lead to better truth-tracking or superior performance in some way.
The primary thing I’m worried about there is when the superior performance is on a metric like “loyalty to the cause” or “not causing problems for ialdabaoth.” On a top-level comment, Isnasene writes:
Which is basically my sense; the general pattern was that ialdabaoth would insist on his frame / interpretation that he was being oppressed by society / the other people in the conversation, and would use a style of negotiation that put the pressure on the other person to figure out how to accommodate him / not be guilty of oppressing him instead of on him to not violate their boundaries. There’s also a general sense that he was aware the of the risk to him for open communication or clear thinking about about him, and thus would work against both when possible.
Thanks for explaining this part; this is really helpful. This model seems to assume that the “oversight” of the “normal system” at the center of the gravity well is trustworthy. I’m currently most worried[1] about the scenario where the “normal system” is corrupt: that new people are getting drawn in by the awesomeness of the Sequences and Harry Potter and the Methods,[2] only to get socialized into a community whose leadership and dominant social trend pays lip service to “rationality”, but is not actually interested in using reasoning (at least, not in public) when using reasoning would be socially inconvenient (whether due to the local area’s political environment, the asymmetric incentives faced by all sufficiently large organizations, the temptation to wirehead on our own “rationality” and “effectiveness” marketing promises, or many other possible reasons) and therefore require a small amount of bravery.
As Michael Vassar put it in 2013:
Have you thought at all about how to prevent the center of the gravity well from becoming predatory? Obviously, I’m all in favor of having systems to catch missing-stair rapists. But if you’re going to build an “immune system” to delegitimize anyone “held in low regard” without having to do the work of engaging with their arguments—without explaining why an amorphous mob that holds something in “low regard” can be trusted to reach that judgement for reliably good reasons—then you’re just running a cult. And if enough people who remember the spirit of the Sequences notice their beloved rationality community getting transformed into a cult, then you might have a rationalist civil war on your hands.
(Um, sorry if that’s too ominous or threatening of a phrasing. I think we mostly want the same thing, but have been following different strategies and exposed to different information, and I notice myself facing an incentive to turn up the rhetoric and point menacingly at my BATNA in case that helps with actually being listened to, because recent experiences have trained my brain to anticipate that even high-ranking “rationalists” are more interested in avoiding social threat than listening to arguments. As I’m sure you can also see, this is already a very bad sign of the mess we’re in.)
“Worried” is an understatement. It’s more like panicking continuously all year with many hours of lost sleep, crying fits, pacing aimlessly instead of doing my dayjob, and eventually doing enough trauma processing to finish writing my forthcoming 20,000-word memoir explaining in detail (as gently and objectively as possible while still telling the truth about my own sentiments and the world I see) why you motherfuckers are being incredibly intellectually dishonest (with respect to a sense of “intellectual dishonesty” that’s about behavior relative to knowledge, not conscious verbal “intent”).
Notably, written at a time Yudkowsky and “the community” had a lower public profile and therefore faced less external social pressure. This is not a coincidence because nothing is ever a coincidence.
On the core point, I think you improve / fix problems with the normal system in the boring, hard ways, and do deeply appreciate you championing particular virtues even when I disagree on where the balance of virtues lies.
I find something offputting here about the word “trustworthy,” because I feel like it’s a 2-place word; I think of oversight as something like “good enough to achieve standard X”, whereas “trustworthy” alone seems to imply there’s a binary standard that is met or not (and has been met). It seems like we could easily have very different standards for trustworthiness that cause us to not disagree on the facts while disagreeing on the implications.
(Somehow, it reminds me of this post and Caledonian’s reaction to it.)
Yes. Mostly this has focused on recruitment work for MIRI, where we really don’t want to guilt people into working on x-risk reduction (as not only is it predatory, it also is a recipe for them burning out instead of being productive, and so morality and efficiency obviously align), and yet most of the naive ways to ask people to consider working on x-risk reduction risk guilting them, and you need a more sophisticated way to remove that failure mode than just saying “please don’t interpret this as me guilting you into it!”. This is a thing that I’ve already written that’s parts of my longer thoughts here.
And, obviously, when I think about moderating LessWrong, I think about how to not become corrupt myself, and what sorts of habits and systems lower the chances of that, or make it more obvious if it does happen.
Right, I agree that we don’t want to get into a pointless pseudo-argument where everyone agrees that x = 60, and yet we have a huge shouting match over whether this should be described using the English word “large” or “small.”
Maybe a question that would lead to a more meaningful disagreement would be, “Should our culture become more or less centralized?”—where centralized is the word I’m choosing to refer to a concept I’m going to try to describe extensionally/ostensively in the following two paragraphs.[1]
A low-centralization culture has slogans like, “Nullis in verba” or “Constant vigilance!”. If a fringe master sets up shop on the outskirts of town, the default presumption is that (time permitting) you should “consider it open-mindedly and then steal only the good parts [...] [as] an obvious guideline for how to do generic optimization”, not because most fringe masters are particularly good (they aren’t), but because thinking for yourself actually works and it’s not like our leaders in the town center know everything already.
In a high-centralization culture, there’s a stronger presumption that our leaders in the town center come closer to knowing everything already, and that the reasoning styles or models being hawked by fringe masters are likely to “contain traps that the people absorbing the model are unable to see”: that is, thinking for yourself doesn’t work. As a result, our leaders might talk up “the value of having a community-wide immune system” so that they can “act against people who are highly manipulative and deceitful before they have clear victims.” If a particular fringe master starts becoming popular, our leaders might want to announce that they are “actively hostile to [the fringe master], and make it clear that [we] do not welcome support from those quarters.”
You seem to be arguing that we should become more centralized. I think that would be moving our culture in the absolute wrong direction. As long as we’re talking about patterns of adversarial optimization, I have to say that, to me, this kind of move looks optimized for “making it easier to ostracize and silence people who could cause trouble for MIRI and CfAR (e.g., Vassar or Ziz), either by being persistent critics or by embarrassing us in front of powerful third parties who are using guilt-by-association heuristics”, rather than improving our collective epistemics.
This seems like a substantial disagreement, rather than a trivial Sorites problem about how to use the word “trustworthy”.
Thanks. I like you, too.
I just made this up, so I’m not at all confident this is the right concept, much like how I didn’t think contextualing-vs.-decoupling was the right concept.
I think the leaders in the town center do not know everything already. I think different areas have different risks when it comes to “thinking for yourself.” It’s one thing to think you can fly and jump off a roof yourself, and another thing to think it’s fine to cook for people when you’re Typhoid Mary, and I worry that you aren’t drawing a distinction here between those cases.
I have thought about this a fair amount, but am not sure I’ve discovered right conceptual lines here, and would be interested in how you would distinguish between the two cases, or if you think they are fundamentally equivalent, or that one of them isn’t real.
In short, I think there are some centralizing moves that are worth it, and others that aren’t, and that we can choose policies individually instead of just throwing the lever on “centralization: Y/N”. Well-Kept Gardens Die by Pacifism is ever relevant; here, the thing that seems relevant to me is that there are some basic functions that need to happen (like, say, the removal of spam), and fulfilling those functions requires tools that could also be used for nefarious functions (as we could just mark criticisms of MIRI as ‘spam’ and they would vanish). But the conceptual categories that people normally have for this are predicated on the interesting cases; sure, both Nazi Germany and WWII America imprisoned rapists, but the interesting imprisonments are of political dissidents, and we might prefer WWII America because it had many fewer such political prisoners, and further prefer a hypothetical America that had no political prisoners. But this spills over into the question of whether we should have prisons or justice systems at all, and I think people’s intuitions on political dissidents are not very useful for what should happen with the more common sort of criminal.
Like, it feels almost silly to have to say this, but I like it when people put forth public positions that are critical of an idea I favor, because then we can argue about it and it’s an opportunity for me to learn something, and I generally expect the audience to be able to follow it and get things right. Like, I disagreed pretty vociferously with The AI Timelines Scam, and yet I thought the discussion it prompted was basically good. It did not ping my Out To Get You sensors in the way that ialdabaoth does. To me, this feels like a central example of the sort of thing you see in a less centralized culture where people are trying to think things through for themselves and end up with different answers, and is not at risk from this sort of moderation.
I don’t think this conversation is going to make any progress at this level of abstraction and in public. I might send you an email.
I look forward to receiving it.
I think I’m someone who might be sympathetic to your case, but just don’t understand what it is, so I’m really curious about this “memoir”. (Let me know if you want me to read your current draft.) Is this guess, i.e., you’re worried about the community falling prey to runaway virtue signaling, remotely close?
Close! I’ll PM you.
This comment of yours does a better job of explaining that than the post.
The separation I’m hoping to make is between banning him because “we know he committed sex crimes” and banning him because “he’s promoting reasoning styles in a way we think is predatory.” We do not know the first to the standard of legal evidence; ialdabaoth has not been convicted in a court of law, and while I think the community investigations were adequate for the questions of whether or not he should be allowed at particular events or clubs, my sense is that his exile was decided by amorphous community-wide processing in a way that I’m reluctant to extend further.
I’m making the additional claim that “The Bay Area community has a rough consensus to kick this guy out” by itself does not meet my bar for banning someone from LessWrong, given the different dynamics of in-person and online interactions. (As a trivial example, suppose someone’s body has a persistent and horrible smell; it could easily be the utilitarian move to not allow them at any physical meetups while giving them full freedom to participate online.) I think this is the bit Tenoke is finding hardest to swallow; it’s one thing to say “yep, this guy is exiled and we’re following the herd” and another thing to say “we’ve exercised independent judgment here, despite obvious pressures to conform.” The latter is a more surprising claim, and correspondingly would require more evidence.
I think this is indirectly true. That is, there’s a separation between expected harm and actual harm, and I’m trying to implement procedures that reduce expected harm. Consider the difference between punishing people for driving drunk and just punishing people for crashing. It’s one thing to just wait until someone accumulates a cloud of ‘unfortunate events’ around them that leads to them finally losing their last defenders, and another to take active steps to assess risks and reduce them. Note that this requires a good model of how ‘drunkenness’ leads to ‘crashes’, and I do not see us as having presented a convincing model of that in this case.
Of course, this post isn’t an example of that; as mentioned, this post is years late, and the real test of whether we can do the equivalent of punishing people for driving drunk is whether we can do anything about people currently causing problems [in expectation]. But my hope is that this community slowly moves from a world where ‘concerns about X’ are published years after they’ve become mutual knowledge among people in the know to one where corrosive forces are actively cleaned up before they make things substantially worse.
“Good influence” in what context? I remember finding some of his Facebook posts/comments insightful, and I’m glad I read them. It shouldn’t be surprising that someone could have some real insights (and thereby constitute a “good influence” in the capacity of social-studies blogging), while also doing a lot of bad things (and thereby constituting a very bad influence in the capacity of being a real-life community member), even if the content of the insights is obviously related to the bad things. (Doing bad things and getting away with them for a long time requires skills that might also lend themselves to good social-studies blogging.)
Less Wrong is in the awkward position of being a public website (anyone can submit blog posts about rationality under a made-up name), and also being closely associated with a real-life community with dense social ties, group houses, money, &c. If our actual moderation algorithm is, “Ban people who have have been justly ostracized from the real-life community as part of their punishment, even if their blog comments were otherwise OK”, that’s fine, but we shouldn’t delude ourselves about what the algorithm is.
If you learned that someone in the rationality community had taken on ialdabaoth as a master (like in the context of zen, or a PhD advisor, or so on), would you expect them to grow in good directions or bad directions? [Ideally, from the epistemic state you were in ~2 years ago, rather than the epistemic state you’re in now.]
I acknowledge that this is quite different from the “would you ever appreciate coming across their thoughts?” question; as it happens, I upvoted Affordance Widths when I first saw it, because it was a neat simple presentation of a model of privilege, and wasn’t taking responsibility for his collected works or thinking about how it fit into them. I mistakenly typical-minded on which parts of his work people were listening to, and which they were safely ignoring.
I agree that algorithm could be fine, for versions of LessWrong that are focused primarily on the community instead of on epistemic progress, but am trying to help instantiate the version of LessWrong that is primarily focused on epistemic progress.
Bad directions. The problem is that I also think this of other users who we aren’t banning, which suggests that our standard for “allowed to post on Less Wrong” is lower than “would be a good master.”
Okay, right, I’m much less sure that I would have said “bad directions” as confidently 2 years ago.
Thanks for this!! (I think I’m much more worried than you about the failure mode where something claiming to make intellectual progress is actually doing something else, which makes me more willing to tolerate pragamatic concessions of principle that are explicitly marked as such, when I’m worried that the alternative is the concession being made anyway with a fake rationale attached.)
I interpreted ialdabaoth as trying to be a master, in a way that I do not interpret most of the people who fail my “would be a good master” check. (Most of them are just not skilled in that sort of thing and don’t seek it out.) If there are people you think would both predictably mislead their students and appear to be trying to recruit students, I’m interested in hearing about it.
This is possible; my suspicion is that we have similar levels of dispreference for that and different models of how attempts to make intellectual progress go astray. I try to be upfront about when I make pragmatic concessions, as I think you’ve have some evidence of, in part so that when I am not making such a concession it’s more believable. [Of course, for someone without observations of that sort of thing, I don’t expect me claiming that I do it to be much evidence.]