So in general I’m noticing a pattern where you make claims about things that happened, but it turns out those things didn’t happen, or there’s no evidence that they happened and no reason one would believe they did a priori, and you’re actually just making an inference and presenting as the state of reality. These seem to universally be inferences which cast other’s motives or actions in a negative light. They seem to be broadly unjustified by the provided evidence and surrounding context, or rely on models of reality (both physical and social) which I think are very likely in conflict with the models held by the people those inferences are about. Sometimes you draw correspondences between the beliefs and/or behaviors of different people or groups, in what seems like an attempt to justify the belief/behavior of the first, or to frame the second as a hypocrite for complaining about the first (though you don’t usually say why these comparisons are relevant). These correspondences turn out to only be superficial similarities while lacking any of the mechanistic similarities that would make them useful for comparisons, or actually conceal the fact that the two parties in question have opposite beliefs on a more relevant axis. For lack of a better framing, it seems like you’re failing to disentangle your social reality from the things that actually happened. I am deeply sympathetic to the issues you experienced, but the way this post (and the previous post) was written makes it extremely difficult to engage with productively, since so many of the relevant claims turn out to not to be claims about things that actually occurred, despite their appearance as such.
I went through the post and picked out some examples (stopping only because it was taking too much of my evening, not because I ran out), with these being the most salient:
> 1. Many others have worked to conceal the circumstances of their deaths
As far as I can tell, this is almost entirely unsubstantiated, with the possible exception of Maia, and in that case it would have been Ziz’s circle doing the concealment, not any of the individuals you express specific concerns about.
> 2. My psychotic break in which I imagined myself creating hell was a natural extension of this line of thought.
The way this is written makes it sound like you think that it ought to have been a (relatively) predictable consequence.
> 3. By the law of excluded middle, the only possible alternative hypothesis is that the problems I experienced at MIRI and CFAR were unique or at least unusually severe, significantly worse than companies like Google for employees’ mental well-being.
In theory, the problems you experienced could have come from sources other than your professional environment. That is a heck of a missing middle.
> 4. This view is rapidly becoming mainstream, validated by research performed by MAPS and at Johns Hopkins, and FDA approval for psychedelic psychotherapy is widely anticipated in the field.
This seems to imply that Michael’s view on the subject corresponds in most relevant ways to the views taken by MAPS/etc. I don’t know what Michael’s views on the subject actually are, but on priors I’m extremely skeptical that the correspondence is sufficient to make this a useful comparison (which, as an appeal to authority, is already on moderately shaky grounds).
> 5. including a report from a friend along the lines of “CFAR can’t legally recommend that you try [a specific psychedelic], but...”
Can you clarify what relationship this friend had with CFAR? This could be concerning if they were a CFAR employee at the time. If they were not a CFAR employee, were they quoting someone who was? If neither, I’m not sure why it’s evidence of CFAR’s views on the subject.
> 7. MIRI leaders were already privately encouraging me to adopt a kind of conflict theory in which many AI organizations were trying to destroy the world
This is not supported by your later descriptions of those interactions.
First, at no point do you describe any encouragement to adopt a conflict-theoretic view. I assume this section is the relevant one: “MIRI leaders including Eliezer Yudkowsky and Nate Soares told me that this was overly naive, that DeepMind would not stop dangerous research even if good reasons for this could be given. Therefore (they said) it was reasonable to develop precursors to AGI in-house to compete with organizations such as DeepMind in terms of developing AGI first. So I was being told to consider people at other AI organizations to be intractably wrong, people who it makes more sense to compete with than to treat as participants in a discourse.” This does not describe encouragement to adopt a conflict-theoretic view. It describes encouragement to adopt some specific beliefs (e.g. about DeepMind’s lack of willingness to integrate information about AI safety into their models and then behave appropriately, and possible ways to mitigate the implied risks), but these are object-level claims, not ontological claims.
Second, this does not describe a stated belief that “AI organizations were trying to destroy the world”. Approximately nobody believes that AI researchers at e.g. DeepMind are actively trying to destroy the world. A more accurate representation of the prevailing belief would be something like “they are doing something that may end up destroying the world, which, from their perspective, would be a totally unintentional and unforeseen consequence of their actions”. This distinction is important, I’m not just nitpicking.
> 8.I was given ridiculous statements and assignments including the claim that MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it, without being given hints.
I would be pretty surprised if this turned out to be an accurate summary of those interactions. In particular, that: 1) MIRI (Nate) believed, as of 2017, that would it be possible to develop AGI given known techniques & technology in 2017, with effectively no new research or breakthroughs required, just implementation, and 2) You were told that you should be able to come up with such a design yourself on short notice without any help or collaborative effort.
Indeed, the anecdote you later relay about your interaction with Nate does not support either of those claims, though it carries its own confusions (why would he encourage you to think about how to develop a working AGI using existing techniques if it would be dangerous to tell you outright? The fact that there’s an extremely obvious contradiction here makes me think that there was a severe miscommunication on at least one side of this conversation).
> 10. His belief that mental states somewhat in the direction of psychosis, such as those had by family members of schizophrenics, are helpful for some forms of intellectual productivity is also shared by Scott Alexander and many academics.
This seems to be (again) drawing a strong correspondence between Michael’s beliefs and actions taken on the basis of those beliefs, and Scott’s beliefs. Scott’s citation of research “showing greater mental modeling and verbal intelligence in relatives of schizophrenics” does not imply that Scott thinks it is a good idea to attempt to induce sub-clinical schizotypal states in people—in fact I would bet a lot of money that Scott thinks doing so is an extremely bad idea, which is a more relevant basis on which to compare his belief’s with Michael’s.
> 11. Scott asserts that Michael Vassar discourages people from seeking mental health treatment. Some mutual friends tried treating me at home for a week as I was losing sleep and becoming increasingly mentally disorganized before deciding to send me to a psychiatric institution, which was a reasonable decision in retrospect.
Were any of those people Michael Vassar? If not, I’m not sure how it’s intended to rebut Scott’s claim (though in general I agree that Scott’s claim about Michael’s discouragement could stand to be substantiated in some way). If so, retracted, but then why is that not specified here, given how clearly it rebuts one of the arguments?
> 13. This is inappropriately enforcing the norms of a minority ideological community as if they were widely accepted professional standards.
This does not read like a charitable interpretation of Scott’s concerns. To wit, if I was friends with someone who I knew was a materialist atheist rationalist, and then one day they came to me and started talking about auras and demons in a way which made it sound like they believed they existed (the way materialist phenomena exist, rather than as “frames”, “analogies”, or “I don’t have a better word for this purely internal mental phenomenon, and this word, despite all the red flags, has connotations that are sufficiently useful that I’m going to use it as a placeholder anyways”), I would update very sharply on them having had a psychotic break (or similar). The relevant reference class for how worrying it is that someone believes something is not what the general public believes, it’s how sharp a departure it is from that person’s previous beliefs (and in what direction—a new, sudden, literal belief in auras and demons is a very common symptom of a certain cluster of mental illnesses!). Note: I am not making any endorsement about the appropriateness of involuntary commitment on the basis of someone suddenly expressing these beliefs. I’m not well-calibrated on the likely distribution of outcomes from doing so.
Moving on from the summary:
> I notice that I have encountered little discussion, public or private, of the conditions of Maia Pasek’s death. To a naive perspective this lack of interest in a dramatic and mysterious death would seem deeply unnatural and extremely surprising, which makes it strong evidence that people are indeed participating in this cover-up.
I do not agree that this is the naive perspective. People, in general, do not enjoy discussing suicide. I have no specific reason to believe that people in the community enjoy this more than average, or expect to get enough value out of it to outweigh the unpleasantness. Unless there is something specifically surprising about a specific suicide that seems relevant to the community more broadly, my default expectation would be that people largely don’t talk about it (the same way they largely don’t talk about most things not relevant to their interests). As far as I can tell, Maia was not a public figure in a way that would, by itself, be sufficient temptation to override people’s generalized dispreference for gossip on the subject. Until today I had not seen any explanation of the specific chain of events preceding Maia’s suicide; a reasonable prior would have been “mentally ill person commits suicide, possibly related to experimental brain hacking they were attempting to do to themselves (as partially detailed on their own blog)”. This seems like fairly strong supporting evidence. I suppose the relevant community interest here is “consider not doing novel neuropsychological research on yourself, especially if you’re already not in a great place”. I agree with that as a useful default heuristic, but it’s one that seems “too obvious to say out loud”. Where do you think a good place for a PSA is?
In general it’s not clear what kind of cover-up you’re imagining. I have not seen any explicit or implicit discouragement of such discussion, except in the (previously mentioned) banal sense that people don’t like discussing suicide and hardly need additional reasons to avoid it.
> While there is a post about Jay’s death on LessWrong, it contains almost no details about Jay’s mental state leading up to their death, and does not link to Jay’s recent blog post. It seems that people other than Jay are also treating the circumstances of Jay’s death as an infohazard.
Jay, like Maia, does not strike me as a public figure. The post you linked is strongly upvoted (+124 at time of writing). It seems to be written as a tribute, which is not the place where I would link to Jay’s most recent blog post if I were trying to analyze causal factors upstream of Jay’s suicide. Again, my prior is that nothing about the lack of discussion needs an explanation in the form of an active conspiracy to suppress such discussion.
> There is a very disturbing possibility (with some evidence for it) here, that people may be picked off one by one (sometimes in ways they cooperate with, e.g. through suicide), with most everyone being too scared to investigate the circumstances.
Please be specific, what evidence? This is an extremely serious claim. For what it’s worth, I don’t agree that people can be picked off in “ways they cooperate with, e.g. though suicide”, unless you mean that Person A would make a concerted effort to convince Person B that Person B ought to commit suicide. But, to the extent that you believe this, the two examples of suicides you listed seem to be causally downstream of “self-hacking with friends(?) in similarly precarious mental states”, not “someone was engaged in a cover-up (of what?) and decided it would be a good idea to try to get these people to commit suicide (why?) despite the incredible risks and unclear benefits”.
> These considerations were widely regarded within MIRI as an important part of AI strategy. I was explicitly expected to think about AI strategy as part of my job. So it isn’t a stretch to say that thinking about extreme AI torture scenarios was part of my job.
S-risk concerns being potentially relevant factors in research does not imply that thinking about specific details of AI torture scenarios would be part of your job. Did someone at MIRI make a claim that imagining S-risk outcomes in graphic detail was necessary or helpful to doing research that factored in S-risks as part of the landscape? Roll to disbelieve.
> Another part of my job was to imagine myself in the role of someone who is going to be creating the AI that could make everything literally the worst it could possibly be, in order to avoid doing that, and prevent others from doing so.
Similarly, roll to disbelieve that someone else at MIRI suggested that you imagine this without significant missing context which would substantially alter the naïve interpretation of that claim.
Skipping the anecdote with Nate & AGI, as I addressed it above, but following that:
> Nate and others who claimed or implied that they had such information did not use it to win bets or make persuasive arguments against people who disagreed with them, but instead used the shared impression or vibe of superior knowledge to invalidate people who disagreed with them.
Putting aside the question of whether Nate or others actually claimed or implied that they had a working model of how to create an AGI with 2017-technology, if they had made such a claim, I am not sure why you would expect them to try to use that model to win bets or make persuasive arguments. I would in fact expect them to never say anything about it to the outside world, because why on earth would you do that given, uh, the entire enterprise of MIRI?
> But I was systematically discouraged from talking with people who doubted that MIRI was for real or publicly revealing evidence that MIRI was not for real, which made it harder for me to seriously entertain that hypothesis.
Can you concretize this? When I read this sentence, an example interaction I can imagine fitting this description would be someone at MIRI advising you in a conversation to avoid talking to Person A, because Person A doubted that MIRI was for real (rather than for other reasons, like “people who spend a lot of time interacting with Person A seem to have a curious habit of undergoing psychotic breaks”).
> In retrospect, I was correct that Nate Soares did not know of a workable AGI design.
I am not sure how either of the excerpts support the claim that “Nate Soares did not know of a workable AGI design” (which, to be clear, I agree with, but for totally unrelated reasons described earlier). Neither of them make any explicit or implicit claims about knowledge (or lack thereof) of AGI design.
> In a recent post, Eliezer Yudkowsky explicitly says that voicing “AGI timelines” is “not great for one’s mental health”, a new additional consideration for suppressing information about timelines.
This is not what Eliezer says. Quoting directly: “What feelings I do have, I worry may be unwise to voice; AGI timelines, in my own experience, are not great for one’s mental health, and I worry that other people seem to have weaker immune systems than even my own.”
This is making a claim that AI timelines themselves are poor for one’s mental health (presumably, the consideration of AI timelines), not the voicing of them.
> Researchers were told not to talk to each other about research, on the basis that some people were working on secret projects and would have to say so if they were asked what they were working on. Instead, we were to talk to Nate Soares, who would connect people who were working on similar projects. I mentioned this to a friend later who considered it a standard cult abuse tactic, of making sure one’s victims don’t talk to each other.
The reason cults attempt to limit communication between their victims is to prevent the formation of common knowledge of specific abusive behaviors that the cult is engaging in, and similar information-theoretic concerns. Taking for granted the description of MIRI’s policy (and application of it) on internal communication about research, this is not a valid correspondence; they were not asking you to avoid discussing your interactions with MIRI (or individuals within MIRI) with other MIRI employees, which could indeed be worrying if it were a sufficiently general ask (rather than e.g. MIRI asking someone in HR not to discuss confidential details of various employees with other employees, which would technically fit the description above but is obviously not what we’re talking about).
> It should be noted that, as I was nominally Nate’s employee, it is consistent with standard business practices for him to prevent me from talking with people who might distract me from my work; this goes to show the continuity between “cults” and “normal corporations”.
I can’t parse this in a way which makes it seem remotely like “standard business practices”. I disagree that it is a standard business practice to actively discourage employees from talking to people who might distract them from their work, largely because employees do not generally have a problem with being distracted from their work because they are talking to specific people. I have worked at number of different companies, each very different from the last in terms of size, domain, organizational culture, etc, and there was not a single occasion where I felt the slightest hint that anyone above me on the org ladder thought I ought to not talk to certain people to avoid distractions, nor did I ever feel like that was part of the organization’s expectations of me.
> MIRI researchers were being very generally denied information (e.g. told not to talk to each other) in a way that makes more sense under a “bad motives” hypothesis than a “good motives” hypothesis. Alternative explanations offered were not persuasive.
Really? This seems to totally disregard explanations unrelated to whether someone has “good motives” or “bad motives”, which are not reasons that I would expect MIRI to have at the top of their list justifying whatever their info-sec policy was.
> By contrast, Michael Vassar thinks that it is common in institutions for people to play zero-sum games in a fractal manner, which makes it unlikely that they could coordinate well enough to cause such large harms.
This is a bit of a sidenote but I’m not sure why this claim is interesting w.r.t. AI alignment, since the problem space, almost by definition, does not require coordination with intent to cause large harms, in order to in fact cause large harms. If the claim is that “institutions are sufficiently dysfunctional that they’ll never be able to build an AGI at all”, that seems like a fully-general argument against institutions ever achieving any goals that require any sort of internal and/or external coordination (trivially invalidated by looking out the window).
> made a medication suggestion (for my sleep issues) that turned out to intensify the psychosis in a way that he might have been able to predict had he thought more carefully
I want to note that while this does not provide much evidence in the way of the claim that Michael Vassar actively seeks to induce psychotic states in people, it is in fact a claim that Michael Vassar was directly, causally upstream of your psychosis worsening, which is worth considering in light of what this entire post seems to be arguing against.
As far as I can tell, this is almost entirely unsubstantiated, with the possible exception of Maia, and in that case it would have been Ziz’s circle doing the concealment, not any of the individuals you express specific concerns about.
I mentioned:
I did it too (there are others like me).
Ziz labeling it as an infohazard is in compliance with feedback Ziz has received from community leaders.
People didn’t draw attention to Fluttershy’s recent blog post, even after I posted about it.
The way this is written makes it sound like you think that it ought to have been a (relatively) predictable consequence.
Whether or not it’s predictable ahead of time, the extended account I give shows the natural progression of thought.
In theory, the problems you experienced could have come from sources other than your professional environment. That is a heck of a missing middle.
Even if there were other causes I still experienced these problems at MIRI. Also, most of the post is an argument that the professional environment contributed quite a lot.
I don’t know what Michael’s views on the subject actually are, but on priors I’m extremely skeptical that the correspondence is sufficient to make this a useful comparison (which, as an appeal to authority, is already on moderately shaky grounds).
Obviously he’s going to disagree with them on specifics, I’m mentioning them as agreeing on the general view Scott attributed to Michael.
Can you clarify what relationship this friend had with CFAR? This could be concerning if they were a CFAR employee at the time. If they were not a CFAR employee, were they quoting someone who was? If neither, I’m not sure why it’s evidence of CFAR’s views on the subject.
Not an employee. Did some teaching at CFAR events. Implied they were telling me information about a CFAR employee’s opinion. Even if they’re repeating a rumor, that still implies use of that psychedelic is common in the social circle, even if that doesn’t mean CFAR caused it at all.
This does not describe encouragement to adopt a conflict-theoretic view. It describes encouragement to adopt some specific beliefs (e.g. about DeepMind’s lack of willingness to integrate information about AI safety into their models and then behave appropriately, and possible ways to mitigate the implied risks), but these are object-level claims, not ontological claims.
The distinction I am making is about (a) treating DeepMind as a participant in discourse, who can be convinced by reasoned argument to do better things, or (b) treating DeepMind as a competitor in a race, who can’t be reasoned with, but has to be beaten decisively in technological capacity. It seems natural to me to label (a) as “mistake theory” and (b) as “conflict theory”, and you have a philosophical disagreement here, but this seems like a quibble.
Indeed, the anecdote you later relay about your interaction with Nate does not support either of those claims, though it carries its own confusions (why would he encourage you to think about how to develop a working AGI using existing techniques if it would be dangerous to tell you outright? The fact that there’s an extremely obvious contradiction here makes me think that there was a severe miscommunication on at least one side of this conversation).
The account I give does support this. His assignment to me would not make sense as an argument of his proposition, that “the pieces to make AGI are already out there and someone just has to put them together”, unless he expected that in completing the assignment I would gain evidence for that proposition. For that to happen, it would have to be an AGI design that is workable (in the sense that with more compute and some fine-tuning, it would actually work); non-workable AGI designs are created all the time in the AI field, and provide no significant evidence about the proposition he was asserting.
Scott’s citation of research “showing greater mental modeling and verbal intelligence in relatives of schizophrenics” does not imply that Scott thinks it is a good idea to attempt to induce sub-clinical schizotypal states in people
Yes, I mentioned disagreement about tradeoffs later in the post.
Were any of those people Michael Vassar?
I believe he was in the chat thread that made the decision, though I’m not sure.
To wit, if I was friends with someone who I knew was a materialist atheist rationalist, and then one day they came to me and started talking about auras and demons in a way which made it sound like they believed they existed (the way materialist phenomena exist, rather than as “frames”, “analogies”, or “I don’t have a better word for this purely internal mental phenomenon, and this word, despite all the red flags, has connotations that are sufficiently useful that I’m going to use it as a placeholder anyways”), I would update very sharply on them having had a psychotic break (or similar).
Later in the thread I give details about what claims I was actually making. Those claims are broadly consistent with materialist ontology.
Unless there is something specifically surprising about a specific suicide that seems relevant to the community more broadly, my default expectation would be that people largely don’t talk about it (the same way they largely don’t talk about most things not relevant to their interests).
It is massively surprising that the suicide was specifically narratized as being caused by inter-hemisphere conflict and precommitments/extortion. Precommitments/extortion are central LessWrong decision theory topics.
In general it’s not clear what kind of cover-up you’re imagining. I have not seen any explicit or implicit discouragement of such discussion, except in the (previously mentioned) banal sense that people don’t like discussing suicide and hardly need additional reasons to avoid it.
I mentioned the specific example that I told a friend about it and then told them not to tell other people. I would guess that Ziz told others who also did this.
Again, my prior is that nothing about the lack of discussion needs an explanation in the form of an active conspiracy to suppress such discussion.
Passive distributed conspiracy, directing attention away from critical details, is sufficient to hide important parts of the world from view.
Please be specific, what evidence? This is an extremely serious claim. For what it’s worth, I don’t agree that people can be picked off in “ways they cooperate with, e.g. though suicide”, unless you mean that Person A would make a concerted effort to convince Person B that Person B ought to commit suicide. But, to the extent that you believe this, the two examples of suicides you listed seem to be causally downstream of “self-hacking with friends(?) in similarly precarious mental states”, not “someone was engaged in a cover-up (of what?) and decided it would be a good idea to try to get these people to commit suicide (why?) despite the incredible risks and unclear benefits”.
Central people made efforts to convince Ziz that she was likely to be “net negative” due to her willingness to generally reveal information. Ziz’s own moral philosophy may imply that many “non-good” people are net negative. It is unsurprising if ideology influenced people towards suicide. The “being picked off” could come from mostly-unconscious motives.
The story of people being unwilling to confirm or deny whether they would investigate if a friend disappeared was very striking at the time, I was surprised how far people would go to avoid investigation.
Did someone at MIRI make a claim that imagining S-risk outcomes in graphic detail was necessary or helpful to doing research that factored in S-risks as part of the landscape?
No. But the magnitude of how bad they are depends on the specific details. I mentioned a blog post that says to imagine being dipped into lava. This was considered a relevant argument about the severity of s-risks.
Similarly, roll to disbelieve that someone else at MIRI suggested that you imagine this without significant missing context which would substantially alter the naïve interpretation of that claim.
It wasn’t specifically suggested. It was implied by the general research landscape presented e.g. Eliezer’s Arbital article.
Putting aside the question of whether Nate or others actually claimed or implied that they had a working model of how to create an AGI with 2017-technology, if they had made such a claim, I am not sure why you would expect them to try to use that model to win bets or make persuasive arguments.
They shouldn’t expect anyone to believe them if they don’t make persuasive arguments or do something similar. If I don’t agree with them that AI is likely to come soon, why would I do research based on that assumption? Presumably he would prefer me to, if in fact AGI is likely to come soon.
This is not what Eliezer says. Quoting directly: “What feelings I do have, I worry may be unwise to voice; AGI timelines, in my own experience, are not great for one’s mental health, and I worry that other people seem to have weaker immune systems than even my own.”
This is making a claim that AI timelines themselves are poor for one’s mental health (presumably, the consideration of AI timelines), not the voicing of them.
He’s clearly giving “AI timelines are bad for one’s mental health” as a reason why his feelings about AI timelines may be unwise to voice.
Taking for granted the description of MIRI’s policy (and application of it) on internal communication about research, this is not a valid correspondence; they were not asking you to avoid discussing your interactions with MIRI (or individuals within MIRI) with other MIRI employees, which could indeed be worrying if it were a sufficiently general ask (rather than e.g. MIRI asking someone in HR not to discuss confidential details of various employees with other employees, which would technically fit the description above but is obviously not what we’re talking about).
Part of our research was decision theory, so talking about decision theory would be one way to talk about the desirability of the security policies. Also, I mentioned Nate and Anna discouraging Michael Vassar from talking to researchers, which fits the pattern of preventing discussion of interactions with MIRI, as he was a notable critic of MIRI at that time.
I disagree that it is a standard business practice to actively discourage employees from talking to people who might distract them from their work, largely because employees do not generally have a problem with being distracted from their work because they are talking to specific people.
I’ve corrected this due to jefftk making the same point.
Really? This seems to totally disregard explanations unrelated to whether someone has “good motives” or “bad motives”, which are not reasons that I would expect MIRI to have at the top of their list justifying whatever their info-sec policy was.
I specifically said that other explanations offered were not convincing. Maybe they would convince you, but they did not convince me.
I want to note that while this does not provide much evidence in the way of the claim that Michael Vassar actively seeks to induce psychotic states in people, it is in fact a claim that Michael Vassar was directly, causally upstream of your psychosis worsening, which is worth considering in light of what this entire post seems to be arguing against.
I did not mean this as an argument against the claim “Michael Vassar seeks to induce psychotic states in people”. I meant his text to Zack as an argument against this claim. It is not perfect evidence, but Scott did not present significant positive evidence for the claim either.
This comment was annoying to engage with because it seemed like you were trying to learn as little new information as possible from the post while finding as many reasons as possible to dismiss the information, despite the post containing lots of information that I would expect people who haven’t been MIRI employees not to know. I think it’s good practice to respond to specific criticisms but this feels like a drag.
It is massively surprising that the suicide was specifically narratized as being caused by inter-hemisphere conflict and precommitments/extortion. Precommitments/extortion are central LessWrong decision theory topics.
It was narratized that way by Ziz, many people having chosen to be skeptical of claims Ziz makes and there was no way to get an independent source.
Also, confict between hemispheres seems to be an important topic on Ziz’s blog (example).
Precommitments/extortion are central LessWrong decision theory topics.
Yes, but I have never seem them in context “one hemisphere extorting the other by precommiting to suicide” on LessWrong. That sounds to me uniquely Zizian.
I appreciate that you took the time to respond to my post in detail. I explained at the top why I had a difficult time engaging productively with your post (i.e. learning from it). I did learn some specific things, such as the claimed sequence of events prior to Maia’s suicide, and Nate’s recent retraction of his earlier public statement on OpenAI. Those are things which are either unambiguously claims about reality, or have legible evidence supporting them.
I mentioned:
I did it too.
Ziz labeling it as an infohazard is in compliance with feedback Ziz has received from community leaders.
People didn’t draw attention to Fluttershy’s recent blog post, even after I posted about it.
None of these carry the same implication that the community, centrally, was engaging in the claimed concealment. This phrasing deflects agency away from the person performing the action, and on to community leaders: “Ziz labeling it as an infohazard is in compliance with feedback Ziz has received from community leaders.” “Ziz labeled something that might have contributed to Maia’s suicide an infohazard, possibly as a result of feedback she got from someone else in the community well before she shared that information with Maia” implies something very different from “Many others have worked to conceal the circumstances of their deaths”, which in context makes it sound like an active conspiracy engaged in by central figures in the community. People not drawing attention to Fluttershy’s post, even after you posted about it, is not active concealment.
> In theory, the problems you experienced could have come from sources other than your professional environment. That is a heck of a missing middle.
Most of the post is an argument that the professional environment contributed quite a lot.
Your original claim included the phrase “the only possible alternative hypothesis”, so this seems totally non-responsive to my problem with it.
> I don’t know what Michael’s views on the subject actually are, but on priors I’m extremely skeptical that the correspondence is sufficient to make this a useful comparison (which, as an appeal to authority, is already on moderately shaky grounds).
Obviously he’s going to disagree with them on specifics, I’m mentioning them as agreeing on the general view Scott attributed to Michael.
Again, this seems non-responsive to what I’m saying is the issue, which is that the “general view” is more or less useless for evaluating how much in “agreement” they really are, as opposed to the specific details.
Not an employee. Did some teaching at CFAR events. Implied they were telling me information about a CFAR employee’s opinion. Even if they’re repeating a rumor, that still implies use of that psychedelic is common in the social circle, even if that doesn’t mean CFAR caused it at all.
That’s good to know, thanks. I think it would make your point here much stronger and more legible if those specific details were included in the original claim.
The distinction I am making is about (a) treating DeepMind as a participant in discourse, who can be convinced by reasoned argument to do better things, or (b) treating DeepMind as a competitor in a race, who can’t be reasoned with, but has to be beaten decisively in technological capacity. It seems natural to me to label (a) as “mistake theory” and (b) as “conflict theory”, and you have a philosophical disagreement here, but this seems like a quibble.
I agree that, as presented, adopting those object-level beliefs would seem to more naturally lend itself to a conflict theory view (vs. mistake theory), but my quibble is with the way you phrased your inference as to what view made most sense to adopt based on the presented facts as a claim that MIRI told you to adopt that view.
The account I give does support this. His assignment to me would not make sense as an argument of his proposition, that “the pieces to make AGI are already out there and someone just has to put them together”, unless he expected that in completing the assignment I would gain evidence for that proposition. For that to happen, it would have to be an AGI design that is workable (in the sense that with more compute and some fine-tuning, it would actually work); non-workable AGI designs are created all the time in the AI field, and provide no significant evidence about the proposition he was asserting.
Here is your summary: “I was given ridiculous statements and assignments including the claim that MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it, without being given hints.”
Here is the anecdote: “I was told, by Nate Soares, that the pieces to make AGI are likely already out there and someone just has to put them together. He did not tell me anything about how to make such an AGI, on the basis that this would be dangerous. Instead, he encouraged me to figure it out for myself, saying it was within my abilities to do so. Now, I am not exactly bad at thinking about AGI; I had, before working at MIRI, gotten a Master’s degree at Stanford studying machine learning, and I had previously helped write a paper about combining probabilistic programming with machine learning. But figuring out how to create an AGI was and is so far beyond my abilities that this was a completely ridiculous expectation.”
Compare: “MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it” with “the pieces to make AGI are likely already out there and someone just has to put them together… he encouraged me to figure it out for myself, saying it was within my abilities to do so”. The first is making several much stronger claims than the second (w.r.t. certainty & specific knowledge of a working design, and their belief in how difficult it would be for you to generate one yourself under what timeframe), in a way that makes MIRI/Nate seem much more unreasonable. If there are specific details omitted from the anecdote that support those stronger claims, I think it would make sense to include them; else we have something resembling a motte & bailey (specific, strong claim, supported by much weaker anecdote which in many possible worlds describes a totally reasonable interaction).
> Scott’s citation of research “showing greater mental modeling and verbal intelligence in relatives of schizophrenics” does not imply that Scott thinks it is a good idea to attempt to induce sub-clinical schizotypal states in people
Yes, I mentioned disagreement about tradeoffs later in the post.
If you agree that Scott & Michael have critical disagreements about the trade-offs in a way that’s relevant to the question at hand—more so than the surface-level agreement Scott’s writing demonstrates—why is this included at all? The implication that one might reasonably be expected to take from “His belief… is also shared by Scott Alexander and many academics.” is, in fact, more false than not.
I believe he was in the chat thread that made the decision, though I’m not sure.
Thanks for clarifying.
Later in the thread I give details about what claims I was actually making. Those claims are broadly consistent with materialist ontology.
Your original claim: “This increases the chance that someone like me could be psychiatrically incarcerated for talking about things that a substantial percentage of the general public (e.g. New Age people and Christians) talk about, and which could be explained in terms that don’t use magical concepts. This is inappropriately enforcing the norms of a minority ideological community as if they were widely accepted professional standards.”
I’m less concerned with the actual claims you were making at the time than with the fact that there is an extremely reasonable explanation for being concerned (in the general case, not in whatever your specific situation was) if a materialist starts taking seriously about auras and demons. That explanation is not that Scott was “enforcing the norms of a minority ideological community”.
It is massively surprising that the suicide was specifically narratized as being caused by inter-hemisphere conflict and precommitments/extortion. Precommitments/extortion are central LessWrong decision theory topics.
Why is this surprising? I agree that it would be surprising if it was narrativized that way by detached outside observers, given that they’d have had no particular reason to construct that sort of narrative, but given the intense focus those in question had on those subjects, as well as those close to them, I’m not sure why you’d be surprised that they ascribed the suicide to the novel self-experimentation they were doing immediately prior to the suicide, which led the person who committed suicide to publicly declare ahead of time that would be why they committed suicide, if they did. That seems like a totally reasonable narrative from their perspective. But, again, none of this information was widely known. I’ve read most of Ziz’s blog and even I didn’t know about the specific sequence of events you described. I’m still not surprised, though, because I place very little credence on the idea that Maia Pasek correctly reasoned her way into killing herself, starting from anything resembling a set of values I’d endorse.
I mentioned the specific example that I told a friend about it and then told them not to tell other people. I would guess that Ziz told others who also did this.
Ok, granted, you and (maybe) Ziz do seem to do the things you’re claiming the community at large does, as central rather than non-central behaviors. This does not justify the implication behind “which makes it strong evidence that people are indeed participating in this cover-up”.
Passive distributed conspiracy, directing attention away from critical details, is sufficient to hide important parts of the world from view.
I don’t even know what this is claiming. Who is directing attention away from what details? How deliberate is it? How do you distinguish this world from one where “people don’t talk about suicide because they didn’t know the person and generally don’t find much value in attempting to psychoanalyze people who committed suicide”?
Central people made efforts to convince Ziz that she was likely to be “net negative” due to her willingness to generally reveal information. Ziz’s own moral philosophy may imply that many “non-good” people are net negative. It is unsurprising if ideology influenced people towards suicide. The “being picked off” could come from mostly-unconscious motives.
I’m not following the chain of logic here. Ziz claims that Anna told her she thought Ziz was likely to be net negative (in the context of AI safety research), after Ziz directly asked her if she thought that. Are you claiming that Anna was sufficiently familiar with the details of Ziz’s ontology (which, afaik, she hadn’t even developed in any detail at that point?) to predict that it might tempt Ziz to commit suicide? Because I’m not seeing how else you get from “Anna answered a question that Ziz asked” to “people may be picked off one by one”.
The story of people being unwilling to confirm or deny whether they would investigate if a friend disappeared was very striking at the time, I was surprised how far people would go to avoid investigation.
I agree that the interaction, as described, sounds quite unusual, mostly due to the extended length where they refused to provide a concrete “yes or no” answer. I would be surprised if many of my friends would sincerely promise to investigate if I disappeared (though I can think of at least one who would), but I would be even more surprised if they refused to tell me whether or not they’d be willing to do so, in such a protracted manner.
No. But the magnitude of how bad they are depends on the specific details. I mentioned a blog post that says to imagine being dipped into lava. This was considered a relevant argument about the severity of s-risks.
I predict that if we anonymously polled MIRI researchers (or AI alignment researchers more broadly), very few of them would endorse “thinking about extreme AI torture scenarios [is] part of my job”, if it carries the implication that they also need to think about those scenarios in explicit detail rather than “many 0s, now put a minus sign in front of them”.
It wasn’t specifically suggested. It was implied by the general research landscape presented e.g. Eliezer’s Arbital article.
So it sounds like you agree that to the extent that it was part of your job “to imagine myself in the role of someone who is going to be creating the AI that could make everything literally the worst it could possibly be”, that was an inference you drew (maybe reasonably!) from the environment, rather than someone at MIRI telling you explicitly that it was part of your job (or strongly implying the same)? Again, to be clear, my issues are with the framing that makes it seem like these are things that other people did or said, rather than with whether or not these were locally valid inferences to be drawing.
They shouldn’t expect anyone to believe them if they don’t make persuasive arguments or do something similar. If I don’t agree with them that AI is likely to come soon, why would I do research based on that assumption? Presumably he would prefer me to, if in fact AGI is likely to come soon.
The original claim you were ridiculing was not that “AI is likely to come soon”.
He’s clearly giving “AI timelines are bad for one’s mental health” as a reason why his feelings about AI timelines may be unwise to voice.
Agreed, but what you said is this: “In a recent post, Eliezer Yudkowsky explicitly says that voicing “AGI timelines” is “not great for one’s mental health”, a new additional consideration for suppressing information about timelines.”
I even agree that you could reasonably interpret what he actually said (as opposed to what you said he said) as a reason to avoid discussing AI timelines, but, crucially, there is no reason to believe he intended that statement to be read that way, and it doesn’t support your claim that you were “constantly encouraged to think very carefully about the negative consequences of publishing anything about AI”. He is providing a reason why he himself does not make much of a habit of discussing AI timelines, and that reason is that he is worried about the mental health of others, not that he thinks discussing it pessimizes for timeline outcomes.
Part of our research was decision theory, so talking about decision theory would be one way to talk about the desirability of the security policies. Also, I mentioned Nate and Anna discouraging Michael Vassar from talking to researchers, which fits the pattern of preventing discussion of interactions with MIRI, as he was a notable critic of MIRI at that time.
This doesn’t seem responsive to my objection, which is the comparison to cult abuse tactics. Asking Michael Vassar to not talk to researchers (during work hours?) is very much not the same thing as asking researchers not to talk to each other. I agree that there are situations where asking Michael Vassar not to talk to researchers would have been inappropriate, but without details on the situation I can’t just nod my head and say, “yep, classic cult tactic”, and in any case this is not what you originally provided as evidence(?) of cult-like behavior.
I’ve corrected this due to jefftk making the same point.
Thanks.
I specifically said that other explanations offered were not convincing. Maybe they would convince you, but they did not convince me.
Ok, so this is just a highly specific claim about the internal motivations of other agents that doesn’t have any supporting evidence—not even what unconvincing arguments they offered.
I did not mean this as an argument against the claim “Michael Vassar seeks to induce psychotic states in people”. I meant his text to Zack as an argument against this claim. It is not perfect evidence, but Scott did not present significant positive evidence for the claim either.
Yes, I agree. My understanding of this post is that it’s substantially devoted to rebutting the argument that Michael Vassar was a meaningful contributor to your mental health problems. I think the fact that Michael Vassar directly interacted with you during the relevant timeframe in a way which you yourself think made things worse is notable, in the sense that for most plausible sets of priors, you should probably be updating upwards on the hypothesis that “spending time around Michael Vassar is more likely lead to psychosis than spending time around most other people”, irrespective of his state of knowledge & motivations at the time.
Not going to respond to all these, a lot seem like nitpicks.
Your original claim included the phrase “the only possible alternative hypothesis”, so this seems totally non-responsive to my problem with it.
My other point was that the problems were still experienced “at MIRI” even if they were caused by other things in the social environment.
That’s good to know, thanks. I think it would make your point here much stronger and more legible if those specific details were included in the original claim.
Edited.
Compare: “MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it” with “the pieces to make AGI are likely already out there and someone just has to put them together… he encouraged me to figure it out for myself, saying it was within my abilities to do so”. The first is making several much stronger claims than the second (w.r.t. certainty & specific knowledge of a working design, and their belief in how difficult it would be for you to generate one yourself under what timeframe), in a way that makes MIRI/Nate seem much more unreasonable.
Nate implied he had already completed the assignment he was giving me.
The assignment wouldn’t provide evidence about whether the pieces to make AGI are already out there unless it was “workable” in the sense that iterative improvement with more compute and theory-light technique iteration would produce AGI.
If you agree that Scott & Michael have critical disagreements about the trade-offs in a way that’s relevant to the question at hand—more so than the surface-level agreement Scott’s writing demonstrates—why is this included at all?
Edited to make it clear that they disagree. The agreement is relevant to place a bound on the scope of what they actually disagree on.
I’m not following the chain of logic here. Ziz claims that Anna told her she thought Ziz was likely to be net negative (in the context of AI safety research), after Ziz directly asked her if she thought that. Are you claiming that Anna was sufficiently familiar with the details of Ziz’s ontology (which, afaik, she hadn’t even developed in any detail at that point?) to predict that it might tempt Ziz to commit suicide?
She might have guessed based on Ziz’s utilitarian futurism (this wouldn’t require knowing many specific details), or might not have been thinking about that consciously. It’s more likely she was trying to control Ziz (she has admitted to generally controlling people around CFAR by e.g. hoarding info). I think my general point is that people are trying to memetically compete with each other in ways that involve labeling others “net negative” in a way that people can very understandably internalize and which would lead to suicide. It’s more like a competition to drive each other insane than one to directly kill each other. A lot of competition (e.g. the kind that would be predicted by evolutionary theory) is subconscious and doesn’t indicate legal responsibility.
Anyway, I edited to make it clearer that many of the influences in question are subconscious and/or memetic.
I predict that if we anonymously polled MIRI researchers (or AI alignment researchers more broadly), very few of them would endorse “thinking about extreme AI torture scenarios [is] part of my job”, if it carries the implication that they also need to think about those scenarios in explicit detail rather than “many 0s, now put a minus sign in front of them”.
I predict that they would say that having some philosophical thoughts about negative utilitarianism and related considerations would be part of their job, and that AI torture scenarios are relevant to that, although perhaps not something they would specifically need to think about.
So it sounds like you agree that to the extent that it was part of your job “to imagine myself in the role of someone who is going to be creating the AI that could make everything literally the worst it could possibly be”, that was an inference you drew (maybe reasonably!) from the environment, rather than someone at MIRI telling you explicitly that it was part of your job (or strongly implying the same)?
Edited to make this clearer.
The original claim you were ridiculing was not that “AI is likely to come soon”.
They’re highly related, having a working AGI design is an argument for short timelines.
He is providing a reason why he himself does not make much of a habit of discussing AI timelines, and that reason is that he is worried about the mental health of others, not that he thinks discussing it pessimizes for timeline outcomes.
Sure, I mentioned it as a consideration other than the consideration I already mentioned about making AI come sooner.
I think the fact that Michael Vassar directly interacted with you during the relevant timeframe in a way which you yourself think made things worse is notable, in the sense that for most plausible sets of priors, you should probably be updating upwards on the hypothesis that “spending time around Michael Vassar is more likely lead to psychosis than spending time around most other people”, irrespective of his state of knowledge & motivations at the time.
I agree it’s weak evidence for that proposition. However the fact that he gave me useful philosophical advice is evidence against that proposition. In total the public info Scott and I have revealed provides very little directional evidence about this proposition.
This seems to be (again) drawing a strong correspondence between Michael’s beliefs and actions taken on the basis of those beliefs, and Scott’s beliefs. Scott’s citation of research “showing greater mental modeling and verbal intelligence in relatives of schizophrenics” does not imply that Scott thinks it is a good idea to attempt to induce sub-clinical schizotypal states in people—in fact I would bet a lot of money that Scott thinks doing so is an extremely bad idea, which is a more relevant basis on which to compare his belief’s with Michael’s.
Michael was accussed in the comment thread of the other post that he seeks out people with who are on the schizophrenic spectrum. Michael to the extend that I know seems to believe that those people have “greater mental modeling and verbal intelligence” and that makes them worth spending time with.
Neither my own conversations with him nor any evidence anyone provided show him to believe that’s a good idea to attempt to induce sub-clinical schizotypal states in people.
So in general I’m noticing a pattern where you make claims about things that happened, but it turns out those things didn’t happen, or there’s no evidence that they happened and no reason one would believe they did a priori, and you’re actually just making an inference and presenting as the state of reality. These seem to universally be inferences which cast other’s motives or actions in a negative light. They seem to be broadly unjustified by the provided evidence and surrounding context, or rely on models of reality (both physical and social) which I think are very likely in conflict with the models held by the people those inferences are about. Sometimes you draw correspondences between the beliefs and/or behaviors of different people or groups, in what seems like an attempt to justify the belief/behavior of the first, or to frame the second as a hypocrite for complaining about the first (though you don’t usually say why these comparisons are relevant). These correspondences turn out to only be superficial similarities while lacking any of the mechanistic similarities that would make them useful for comparisons, or actually conceal the fact that the two parties in question have opposite beliefs on a more relevant axis. For lack of a better framing, it seems like you’re failing to disentangle your social reality from the things that actually happened. I am deeply sympathetic to the issues you experienced, but the way this post (and the previous post) was written makes it extremely difficult to engage with productively, since so many of the relevant claims turn out to not to be claims about things that actually occurred, despite their appearance as such.
I went through the post and picked out some examples (stopping only because it was taking too much of my evening, not because I ran out), with these being the most salient:
> 1. Many others have worked to conceal the circumstances of their deaths
As far as I can tell, this is almost entirely unsubstantiated, with the possible exception of Maia, and in that case it would have been Ziz’s circle doing the concealment, not any of the individuals you express specific concerns about.
> 2. My psychotic break in which I imagined myself creating hell was a natural extension of this line of thought.
The way this is written makes it sound like you think that it ought to have been a (relatively) predictable consequence.
> 3. By the law of excluded middle, the only possible alternative hypothesis is that the problems I experienced at MIRI and CFAR were unique or at least unusually severe, significantly worse than companies like Google for employees’ mental well-being.
In theory, the problems you experienced could have come from sources other than your professional environment. That is a heck of a missing middle.
> 4. This view is rapidly becoming mainstream, validated by research performed by MAPS and at Johns Hopkins, and FDA approval for psychedelic psychotherapy is widely anticipated in the field.
This seems to imply that Michael’s view on the subject corresponds in most relevant ways to the views taken by MAPS/etc. I don’t know what Michael’s views on the subject actually are, but on priors I’m extremely skeptical that the correspondence is sufficient to make this a useful comparison (which, as an appeal to authority, is already on moderately shaky grounds).
> 5. including a report from a friend along the lines of “CFAR can’t legally recommend that you try [a specific psychedelic], but...”
Can you clarify what relationship this friend had with CFAR? This could be concerning if they were a CFAR employee at the time. If they were not a CFAR employee, were they quoting someone who was? If neither, I’m not sure why it’s evidence of CFAR’s views on the subject.
> 7. MIRI leaders were already privately encouraging me to adopt a kind of conflict theory in which many AI organizations were trying to destroy the world
This is not supported by your later descriptions of those interactions.
First, at no point do you describe any encouragement to adopt a conflict-theoretic view. I assume this section is the relevant one: “MIRI leaders including Eliezer Yudkowsky and Nate Soares told me that this was overly naive, that DeepMind would not stop dangerous research even if good reasons for this could be given. Therefore (they said) it was reasonable to develop precursors to AGI in-house to compete with organizations such as DeepMind in terms of developing AGI first. So I was being told to consider people at other AI organizations to be intractably wrong, people who it makes more sense to compete with than to treat as participants in a discourse.” This does not describe encouragement to adopt a conflict-theoretic view. It describes encouragement to adopt some specific beliefs (e.g. about DeepMind’s lack of willingness to integrate information about AI safety into their models and then behave appropriately, and possible ways to mitigate the implied risks), but these are object-level claims, not ontological claims.
Second, this does not describe a stated belief that “AI organizations were trying to destroy the world”. Approximately nobody believes that AI researchers at e.g. DeepMind are actively trying to destroy the world. A more accurate representation of the prevailing belief would be something like “they are doing something that may end up destroying the world, which, from their perspective, would be a totally unintentional and unforeseen consequence of their actions”. This distinction is important, I’m not just nitpicking.
> 8.I was given ridiculous statements and assignments including the claim that MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it, without being given hints.
I would be pretty surprised if this turned out to be an accurate summary of those interactions. In particular, that:
1) MIRI (Nate) believed, as of 2017, that would it be possible to develop AGI given known techniques & technology in 2017, with effectively no new research or breakthroughs required, just implementation, and
2) You were told that you should be able to come up with such a design yourself on short notice without any help or collaborative effort.
Indeed, the anecdote you later relay about your interaction with Nate does not support either of those claims, though it carries its own confusions (why would he encourage you to think about how to develop a working AGI using existing techniques if it would be dangerous to tell you outright? The fact that there’s an extremely obvious contradiction here makes me think that there was a severe miscommunication on at least one side of this conversation).
> 10. His belief that mental states somewhat in the direction of psychosis, such as those had by family members of schizophrenics, are helpful for some forms of intellectual productivity is also shared by Scott Alexander and many academics.
This seems to be (again) drawing a strong correspondence between Michael’s beliefs and actions taken on the basis of those beliefs, and Scott’s beliefs. Scott’s citation of research “showing greater mental modeling and verbal intelligence in relatives of schizophrenics” does not imply that Scott thinks it is a good idea to attempt to induce sub-clinical schizotypal states in people—in fact I would bet a lot of money that Scott thinks doing so is an extremely bad idea, which is a more relevant basis on which to compare his belief’s with Michael’s.
> 11. Scott asserts that Michael Vassar discourages people from seeking mental health treatment. Some mutual friends tried treating me at home for a week as I was losing sleep and becoming increasingly mentally disorganized before deciding to send me to a psychiatric institution, which was a reasonable decision in retrospect.
Were any of those people Michael Vassar? If not, I’m not sure how it’s intended to rebut Scott’s claim (though in general I agree that Scott’s claim about Michael’s discouragement could stand to be substantiated in some way). If so, retracted, but then why is that not specified here, given how clearly it rebuts one of the arguments?
> 13. This is inappropriately enforcing the norms of a minority ideological community as if they were widely accepted professional standards.
This does not read like a charitable interpretation of Scott’s concerns. To wit, if I was friends with someone who I knew was a materialist atheist rationalist, and then one day they came to me and started talking about auras and demons in a way which made it sound like they believed they existed (the way materialist phenomena exist, rather than as “frames”, “analogies”, or “I don’t have a better word for this purely internal mental phenomenon, and this word, despite all the red flags, has connotations that are sufficiently useful that I’m going to use it as a placeholder anyways”), I would update very sharply on them having had a psychotic break (or similar). The relevant reference class for how worrying it is that someone believes something is not what the general public believes, it’s how sharp a departure it is from that person’s previous beliefs (and in what direction—a new, sudden, literal belief in auras and demons is a very common symptom of a certain cluster of mental illnesses!). Note: I am not making any endorsement about the appropriateness of involuntary commitment on the basis of someone suddenly expressing these beliefs. I’m not well-calibrated on the likely distribution of outcomes from doing so.
Moving on from the summary:
> I notice that I have encountered little discussion, public or private, of the conditions of Maia Pasek’s death. To a naive perspective this lack of interest in a dramatic and mysterious death would seem deeply unnatural and extremely surprising, which makes it strong evidence that people are indeed participating in this cover-up.
I do not agree that this is the naive perspective. People, in general, do not enjoy discussing suicide. I have no specific reason to believe that people in the community enjoy this more than average, or expect to get enough value out of it to outweigh the unpleasantness. Unless there is something specifically surprising about a specific suicide that seems relevant to the community more broadly, my default expectation would be that people largely don’t talk about it (the same way they largely don’t talk about most things not relevant to their interests). As far as I can tell, Maia was not a public figure in a way that would, by itself, be sufficient temptation to override people’s generalized dispreference for gossip on the subject. Until today I had not seen any explanation of the specific chain of events preceding Maia’s suicide; a reasonable prior would have been “mentally ill person commits suicide, possibly related to experimental brain hacking they were attempting to do to themselves (as partially detailed on their own blog)”. This seems like fairly strong supporting evidence. I suppose the relevant community interest here is “consider not doing novel neuropsychological research on yourself, especially if you’re already not in a great place”. I agree with that as a useful default heuristic, but it’s one that seems “too obvious to say out loud”. Where do you think a good place for a PSA is?
In general it’s not clear what kind of cover-up you’re imagining. I have not seen any explicit or implicit discouragement of such discussion, except in the (previously mentioned) banal sense that people don’t like discussing suicide and hardly need additional reasons to avoid it.
> While there is a post about Jay’s death on LessWrong, it contains almost no details about Jay’s mental state leading up to their death, and does not link to Jay’s recent blog post. It seems that people other than Jay are also treating the circumstances of Jay’s death as an infohazard.
Jay, like Maia, does not strike me as a public figure. The post you linked is strongly upvoted (+124 at time of writing). It seems to be written as a tribute, which is not the place where I would link to Jay’s most recent blog post if I were trying to analyze causal factors upstream of Jay’s suicide. Again, my prior is that nothing about the lack of discussion needs an explanation in the form of an active conspiracy to suppress such discussion.
> There is a very disturbing possibility (with some evidence for it) here, that people may be picked off one by one (sometimes in ways they cooperate with, e.g. through suicide), with most everyone being too scared to investigate the circumstances.
Please be specific, what evidence? This is an extremely serious claim. For what it’s worth, I don’t agree that people can be picked off in “ways they cooperate with, e.g. though suicide”, unless you mean that Person A would make a concerted effort to convince Person B that Person B ought to commit suicide. But, to the extent that you believe this, the two examples of suicides you listed seem to be causally downstream of “self-hacking with friends(?) in similarly precarious mental states”, not “someone was engaged in a cover-up (of what?) and decided it would be a good idea to try to get these people to commit suicide (why?) despite the incredible risks and unclear benefits”.
> These considerations were widely regarded within MIRI as an important part of AI strategy. I was explicitly expected to think about AI strategy as part of my job. So it isn’t a stretch to say that thinking about extreme AI torture scenarios was part of my job.
S-risk concerns being potentially relevant factors in research does not imply that thinking about specific details of AI torture scenarios would be part of your job. Did someone at MIRI make a claim that imagining S-risk outcomes in graphic detail was necessary or helpful to doing research that factored in S-risks as part of the landscape? Roll to disbelieve.
> Another part of my job was to imagine myself in the role of someone who is going to be creating the AI that could make everything literally the worst it could possibly be, in order to avoid doing that, and prevent others from doing so.
Similarly, roll to disbelieve that someone else at MIRI suggested that you imagine this without significant missing context which would substantially alter the naïve interpretation of that claim.
Skipping the anecdote with Nate & AGI, as I addressed it above, but following that:
> Nate and others who claimed or implied that they had such information did not use it to win bets or make persuasive arguments against people who disagreed with them, but instead used the shared impression or vibe of superior knowledge to invalidate people who disagreed with them.
Putting aside the question of whether Nate or others actually claimed or implied that they had a working model of how to create an AGI with 2017-technology, if they had made such a claim, I am not sure why you would expect them to try to use that model to win bets or make persuasive arguments. I would in fact expect them to never say anything about it to the outside world, because why on earth would you do that given, uh, the entire enterprise of MIRI?
> But I was systematically discouraged from talking with people who doubted that MIRI was for real or publicly revealing evidence that MIRI was not for real, which made it harder for me to seriously entertain that hypothesis.
Can you concretize this? When I read this sentence, an example interaction I can imagine fitting this description would be someone at MIRI advising you in a conversation to avoid talking to Person A, because Person A doubted that MIRI was for real (rather than for other reasons, like “people who spend a lot of time interacting with Person A seem to have a curious habit of undergoing psychotic breaks”).
> In retrospect, I was correct that Nate Soares did not know of a workable AGI design.
I am not sure how either of the excerpts support the claim that “Nate Soares did not know of a workable AGI design” (which, to be clear, I agree with, but for totally unrelated reasons described earlier). Neither of them make any explicit or implicit claims about knowledge (or lack thereof) of AGI design.
> In a recent post, Eliezer Yudkowsky explicitly says that voicing “AGI timelines” is “not great for one’s mental health”, a new additional consideration for suppressing information about timelines.
This is not what Eliezer says. Quoting directly: “What feelings I do have, I worry may be unwise to voice; AGI timelines, in my own experience, are not great for one’s mental health, and I worry that other people seem to have weaker immune systems than even my own.”
This is making a claim that AI timelines themselves are poor for one’s mental health (presumably, the consideration of AI timelines), not the voicing of them.
> Researchers were told not to talk to each other about research, on the basis that some people were working on secret projects and would have to say so if they were asked what they were working on. Instead, we were to talk to Nate Soares, who would connect people who were working on similar projects. I mentioned this to a friend later who considered it a standard cult abuse tactic, of making sure one’s victims don’t talk to each other.
The reason cults attempt to limit communication between their victims is to prevent the formation of common knowledge of specific abusive behaviors that the cult is engaging in, and similar information-theoretic concerns. Taking for granted the description of MIRI’s policy (and application of it) on internal communication about research, this is not a valid correspondence; they were not asking you to avoid discussing your interactions with MIRI (or individuals within MIRI) with other MIRI employees, which could indeed be worrying if it were a sufficiently general ask (rather than e.g. MIRI asking someone in HR not to discuss confidential details of various employees with other employees, which would technically fit the description above but is obviously not what we’re talking about).
> It should be noted that, as I was nominally Nate’s employee, it is consistent with standard business practices for him to prevent me from talking with people who might distract me from my work; this goes to show the continuity between “cults” and “normal corporations”.
I can’t parse this in a way which makes it seem remotely like “standard business practices”. I disagree that it is a standard business practice to actively discourage employees from talking to people who might distract them from their work, largely because employees do not generally have a problem with being distracted from their work because they are talking to specific people. I have worked at number of different companies, each very different from the last in terms of size, domain, organizational culture, etc, and there was not a single occasion where I felt the slightest hint that anyone above me on the org ladder thought I ought to not talk to certain people to avoid distractions, nor did I ever feel like that was part of the organization’s expectations of me.
> MIRI researchers were being very generally denied information (e.g. told not to talk to each other) in a way that makes more sense under a “bad motives” hypothesis than a “good motives” hypothesis. Alternative explanations offered were not persuasive.
Really? This seems to totally disregard explanations unrelated to whether someone has “good motives” or “bad motives”, which are not reasons that I would expect MIRI to have at the top of their list justifying whatever their info-sec policy was.
> By contrast, Michael Vassar thinks that it is common in institutions for people to play zero-sum games in a fractal manner, which makes it unlikely that they could coordinate well enough to cause such large harms.
This is a bit of a sidenote but I’m not sure why this claim is interesting w.r.t. AI alignment, since the problem space, almost by definition, does not require coordination with intent to cause large harms, in order to in fact cause large harms. If the claim is that “institutions are sufficiently dysfunctional that they’ll never be able to build an AGI at all”, that seems like a fully-general argument against institutions ever achieving any goals that require any sort of internal and/or external coordination (trivially invalidated by looking out the window).
> made a medication suggestion (for my sleep issues) that turned out to intensify the psychosis in a way that he might have been able to predict had he thought more carefully
I want to note that while this does not provide much evidence in the way of the claim that Michael Vassar actively seeks to induce psychotic states in people, it is in fact a claim that Michael Vassar was directly, causally upstream of your psychosis worsening, which is worth considering in light of what this entire post seems to be arguing against.
I mentioned:
I did it too (there are others like me).
Ziz labeling it as an infohazard is in compliance with feedback Ziz has received from community leaders.
People didn’t draw attention to Fluttershy’s recent blog post, even after I posted about it.
Whether or not it’s predictable ahead of time, the extended account I give shows the natural progression of thought.
Even if there were other causes I still experienced these problems at MIRI. Also, most of the post is an argument that the professional environment contributed quite a lot.
Obviously he’s going to disagree with them on specifics, I’m mentioning them as agreeing on the general view Scott attributed to Michael.
Not an employee. Did some teaching at CFAR events. Implied they were telling me information about a CFAR employee’s opinion. Even if they’re repeating a rumor, that still implies use of that psychedelic is common in the social circle, even if that doesn’t mean CFAR caused it at all.
The distinction I am making is about (a) treating DeepMind as a participant in discourse, who can be convinced by reasoned argument to do better things, or (b) treating DeepMind as a competitor in a race, who can’t be reasoned with, but has to be beaten decisively in technological capacity. It seems natural to me to label (a) as “mistake theory” and (b) as “conflict theory”, and you have a philosophical disagreement here, but this seems like a quibble.
The account I give does support this. His assignment to me would not make sense as an argument of his proposition, that “the pieces to make AGI are already out there and someone just has to put them together”, unless he expected that in completing the assignment I would gain evidence for that proposition. For that to happen, it would have to be an AGI design that is workable (in the sense that with more compute and some fine-tuning, it would actually work); non-workable AGI designs are created all the time in the AI field, and provide no significant evidence about the proposition he was asserting.
Yes, I mentioned disagreement about tradeoffs later in the post.
I believe he was in the chat thread that made the decision, though I’m not sure.
Later in the thread I give details about what claims I was actually making. Those claims are broadly consistent with materialist ontology.
It is massively surprising that the suicide was specifically narratized as being caused by inter-hemisphere conflict and precommitments/extortion. Precommitments/extortion are central LessWrong decision theory topics.
I mentioned the specific example that I told a friend about it and then told them not to tell other people. I would guess that Ziz told others who also did this.
Passive distributed conspiracy, directing attention away from critical details, is sufficient to hide important parts of the world from view.
Central people made efforts to convince Ziz that she was likely to be “net negative” due to her willingness to generally reveal information. Ziz’s own moral philosophy may imply that many “non-good” people are net negative. It is unsurprising if ideology influenced people towards suicide. The “being picked off” could come from mostly-unconscious motives.
The story of people being unwilling to confirm or deny whether they would investigate if a friend disappeared was very striking at the time, I was surprised how far people would go to avoid investigation.
No. But the magnitude of how bad they are depends on the specific details. I mentioned a blog post that says to imagine being dipped into lava. This was considered a relevant argument about the severity of s-risks.
It wasn’t specifically suggested. It was implied by the general research landscape presented e.g. Eliezer’s Arbital article.
They shouldn’t expect anyone to believe them if they don’t make persuasive arguments or do something similar. If I don’t agree with them that AI is likely to come soon, why would I do research based on that assumption? Presumably he would prefer me to, if in fact AGI is likely to come soon.
He’s clearly giving “AI timelines are bad for one’s mental health” as a reason why his feelings about AI timelines may be unwise to voice.
Part of our research was decision theory, so talking about decision theory would be one way to talk about the desirability of the security policies. Also, I mentioned Nate and Anna discouraging Michael Vassar from talking to researchers, which fits the pattern of preventing discussion of interactions with MIRI, as he was a notable critic of MIRI at that time.
I’ve corrected this due to jefftk making the same point.
I specifically said that other explanations offered were not convincing. Maybe they would convince you, but they did not convince me.
I did not mean this as an argument against the claim “Michael Vassar seeks to induce psychotic states in people”. I meant his text to Zack as an argument against this claim. It is not perfect evidence, but Scott did not present significant positive evidence for the claim either.
This comment was annoying to engage with because it seemed like you were trying to learn as little new information as possible from the post while finding as many reasons as possible to dismiss the information, despite the post containing lots of information that I would expect people who haven’t been MIRI employees not to know. I think it’s good practice to respond to specific criticisms but this feels like a drag.
It was narratized that way by Ziz, many people having chosen to be skeptical of claims Ziz makes and there was no way to get an independent source.
Also, confict between hemispheres seems to be an important topic on Ziz’s blog (example).
Yes, but I have never seem them in context “one hemisphere extorting the other by precommiting to suicide” on LessWrong. That sounds to me uniquely Zizian.
I appreciate that you took the time to respond to my post in detail. I explained at the top why I had a difficult time engaging productively with your post (i.e. learning from it). I did learn some specific things, such as the claimed sequence of events prior to Maia’s suicide, and Nate’s recent retraction of his earlier public statement on OpenAI. Those are things which are either unambiguously claims about reality, or have legible evidence supporting them.
None of these carry the same implication that the community, centrally, was engaging in the claimed concealment. This phrasing deflects agency away from the person performing the action, and on to community leaders: “Ziz labeling it as an infohazard is in compliance with feedback Ziz has received from community leaders.” “Ziz labeled something that might have contributed to Maia’s suicide an infohazard, possibly as a result of feedback she got from someone else in the community well before she shared that information with Maia” implies something very different from “Many others have worked to conceal the circumstances of their deaths”, which in context makes it sound like an active conspiracy engaged in by central figures in the community. People not drawing attention to Fluttershy’s post, even after you posted about it, is not active concealment.
Your original claim included the phrase “the only possible alternative hypothesis”, so this seems totally non-responsive to my problem with it.
Again, this seems non-responsive to what I’m saying is the issue, which is that the “general view” is more or less useless for evaluating how much in “agreement” they really are, as opposed to the specific details.
That’s good to know, thanks. I think it would make your point here much stronger and more legible if those specific details were included in the original claim.
I agree that, as presented, adopting those object-level beliefs would seem to more naturally lend itself to a conflict theory view (vs. mistake theory), but my quibble is with the way you phrased your inference as to what view made most sense to adopt based on the presented facts as a claim that MIRI told you to adopt that view.
Here is your summary: “I was given ridiculous statements and assignments including the claim that MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it, without being given hints.”
Here is the anecdote: “I was told, by Nate Soares, that the pieces to make AGI are likely already out there and someone just has to put them together. He did not tell me anything about how to make such an AGI, on the basis that this would be dangerous. Instead, he encouraged me to figure it out for myself, saying it was within my abilities to do so. Now, I am not exactly bad at thinking about AGI; I had, before working at MIRI, gotten a Master’s degree at Stanford studying machine learning, and I had previously helped write a paper about combining probabilistic programming with machine learning. But figuring out how to create an AGI was and is so far beyond my abilities that this was a completely ridiculous expectation.”
Compare: “MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it” with “the pieces to make AGI are likely already out there and someone just has to put them together… he encouraged me to figure it out for myself, saying it was within my abilities to do so”. The first is making several much stronger claims than the second (w.r.t. certainty & specific knowledge of a working design, and their belief in how difficult it would be for you to generate one yourself under what timeframe), in a way that makes MIRI/Nate seem much more unreasonable. If there are specific details omitted from the anecdote that support those stronger claims, I think it would make sense to include them; else we have something resembling a motte & bailey (specific, strong claim, supported by much weaker anecdote which in many possible worlds describes a totally reasonable interaction).
If you agree that Scott & Michael have critical disagreements about the trade-offs in a way that’s relevant to the question at hand—more so than the surface-level agreement Scott’s writing demonstrates—why is this included at all? The implication that one might reasonably be expected to take from “His belief… is also shared by Scott Alexander and many academics.” is, in fact, more false than not.
Thanks for clarifying.
Your original claim: “This increases the chance that someone like me could be psychiatrically incarcerated for talking about things that a substantial percentage of the general public (e.g. New Age people and Christians) talk about, and which could be explained in terms that don’t use magical concepts. This is inappropriately enforcing the norms of a minority ideological community as if they were widely accepted professional standards.”
I’m less concerned with the actual claims you were making at the time than with the fact that there is an extremely reasonable explanation for being concerned (in the general case, not in whatever your specific situation was) if a materialist starts taking seriously about auras and demons. That explanation is not that Scott was “enforcing the norms of a minority ideological community”.
Why is this surprising? I agree that it would be surprising if it was narrativized that way by detached outside observers, given that they’d have had no particular reason to construct that sort of narrative, but given the intense focus those in question had on those subjects, as well as those close to them, I’m not sure why you’d be surprised that they ascribed the suicide to the novel self-experimentation they were doing immediately prior to the suicide, which led the person who committed suicide to publicly declare ahead of time that would be why they committed suicide, if they did. That seems like a totally reasonable narrative from their perspective. But, again, none of this information was widely known. I’ve read most of Ziz’s blog and even I didn’t know about the specific sequence of events you described. I’m still not surprised, though, because I place very little credence on the idea that Maia Pasek correctly reasoned her way into killing herself, starting from anything resembling a set of values I’d endorse.
Ok, granted, you and (maybe) Ziz do seem to do the things you’re claiming the community at large does, as central rather than non-central behaviors. This does not justify the implication behind “which makes it strong evidence that people are indeed participating in this cover-up”.
I don’t even know what this is claiming. Who is directing attention away from what details? How deliberate is it? How do you distinguish this world from one where “people don’t talk about suicide because they didn’t know the person and generally don’t find much value in attempting to psychoanalyze people who committed suicide”?
I’m not following the chain of logic here. Ziz claims that Anna told her she thought Ziz was likely to be net negative (in the context of AI safety research), after Ziz directly asked her if she thought that. Are you claiming that Anna was sufficiently familiar with the details of Ziz’s ontology (which, afaik, she hadn’t even developed in any detail at that point?) to predict that it might tempt Ziz to commit suicide? Because I’m not seeing how else you get from “Anna answered a question that Ziz asked” to “people may be picked off one by one”.
I agree that the interaction, as described, sounds quite unusual, mostly due to the extended length where they refused to provide a concrete “yes or no” answer. I would be surprised if many of my friends would sincerely promise to investigate if I disappeared (though I can think of at least one who would), but I would be even more surprised if they refused to tell me whether or not they’d be willing to do so, in such a protracted manner.
I predict that if we anonymously polled MIRI researchers (or AI alignment researchers more broadly), very few of them would endorse “thinking about extreme AI torture scenarios [is] part of my job”, if it carries the implication that they also need to think about those scenarios in explicit detail rather than “many 0s, now put a minus sign in front of them”.
So it sounds like you agree that to the extent that it was part of your job “to imagine myself in the role of someone who is going to be creating the AI that could make everything literally the worst it could possibly be”, that was an inference you drew (maybe reasonably!) from the environment, rather than someone at MIRI telling you explicitly that it was part of your job (or strongly implying the same)? Again, to be clear, my issues are with the framing that makes it seem like these are things that other people did or said, rather than with whether or not these were locally valid inferences to be drawing.
The original claim you were ridiculing was not that “AI is likely to come soon”.
Agreed, but what you said is this: “In a recent post, Eliezer Yudkowsky explicitly says that voicing “AGI timelines” is “not great for one’s mental health”, a new additional consideration for suppressing information about timelines.”
I even agree that you could reasonably interpret what he actually said (as opposed to what you said he said) as a reason to avoid discussing AI timelines, but, crucially, there is no reason to believe he intended that statement to be read that way, and it doesn’t support your claim that you were “constantly encouraged to think very carefully about the negative consequences of publishing anything about AI”. He is providing a reason why he himself does not make much of a habit of discussing AI timelines, and that reason is that he is worried about the mental health of others, not that he thinks discussing it pessimizes for timeline outcomes.
This doesn’t seem responsive to my objection, which is the comparison to cult abuse tactics. Asking Michael Vassar to not talk to researchers (during work hours?) is very much not the same thing as asking researchers not to talk to each other. I agree that there are situations where asking Michael Vassar not to talk to researchers would have been inappropriate, but without details on the situation I can’t just nod my head and say, “yep, classic cult tactic”, and in any case this is not what you originally provided as evidence(?) of cult-like behavior.
Thanks.
Ok, so this is just a highly specific claim about the internal motivations of other agents that doesn’t have any supporting evidence—not even what unconvincing arguments they offered.
Yes, I agree. My understanding of this post is that it’s substantially devoted to rebutting the argument that Michael Vassar was a meaningful contributor to your mental health problems. I think the fact that Michael Vassar directly interacted with you during the relevant timeframe in a way which you yourself think made things worse is notable, in the sense that for most plausible sets of priors, you should probably be updating upwards on the hypothesis that “spending time around Michael Vassar is more likely lead to psychosis than spending time around most other people”, irrespective of his state of knowledge & motivations at the time.
Not going to respond to all these, a lot seem like nitpicks.
My other point was that the problems were still experienced “at MIRI” even if they were caused by other things in the social environment.
Edited.
Nate implied he had already completed the assignment he was giving me.
The assignment wouldn’t provide evidence about whether the pieces to make AGI are already out there unless it was “workable” in the sense that iterative improvement with more compute and theory-light technique iteration would produce AGI.
Edited to make it clear that they disagree. The agreement is relevant to place a bound on the scope of what they actually disagree on.
She might have guessed based on Ziz’s utilitarian futurism (this wouldn’t require knowing many specific details), or might not have been thinking about that consciously. It’s more likely she was trying to control Ziz (she has admitted to generally controlling people around CFAR by e.g. hoarding info). I think my general point is that people are trying to memetically compete with each other in ways that involve labeling others “net negative” in a way that people can very understandably internalize and which would lead to suicide. It’s more like a competition to drive each other insane than one to directly kill each other. A lot of competition (e.g. the kind that would be predicted by evolutionary theory) is subconscious and doesn’t indicate legal responsibility.
Anyway, I edited to make it clearer that many of the influences in question are subconscious and/or memetic.
I predict that they would say that having some philosophical thoughts about negative utilitarianism and related considerations would be part of their job, and that AI torture scenarios are relevant to that, although perhaps not something they would specifically need to think about.
Edited to make this clearer.
They’re highly related, having a working AGI design is an argument for short timelines.
Sure, I mentioned it as a consideration other than the consideration I already mentioned about making AI come sooner.
I agree it’s weak evidence for that proposition. However the fact that he gave me useful philosophical advice is evidence against that proposition. In total the public info Scott and I have revealed provides very little directional evidence about this proposition.
Michael was accussed in the comment thread of the other post that he seeks out people with who are on the schizophrenic spectrum. Michael to the extend that I know seems to believe that those people have “greater mental modeling and verbal intelligence” and that makes them worth spending time with.
Neither my own conversations with him nor any evidence anyone provided show him to believe that’s a good idea to attempt to induce sub-clinical schizotypal states in people.