I really liked MIRI/CFAR during 2015-2016 (even though I had lots of criticisms), I think I benefited a lot overall, I think things got bad in 2017 and haven’t recovered. E.g. MIRI has had many fewer good publications since 2017 and for reasons I’ve expressed, I don’t believe their private research is comparably good to their previous public research. (Maybe to some extent I got disillusioned so I’m overestimating how much things changed, I’m not entirely sure how to disentangle.)
As revealed in my posts, I was a “dissident” during 2017 and confusedly/fearfully trying to learn and share critiques, gather people into a splinter group, etc, so there’s somewhat of a legacy of a past conflict affecting the present, although it’s obviously less intense now, especially after I can write about it.
I’ve noticed people trying to “center” everything around MIRI, justifying their actions in terms of “helping MIRI” etc (one LW mod told me and others in 2018 that LessWrong was primarily a recruiting funnel for MIRI, not a rationality promotion website, and someone else who was in the scene 2016-2017 corroborated that this is a common opinion), and I think this is pretty bad since they have no way of checking how useful MIRI’s work is, and there’s a market for lemons (compare EA arguments against donating to even “reputable” organizations like UNICEF). It resembles idol worship and that’s disappointing.
This is corroborated by some other former MIRI employees, e.g. someone who left sometime in the past 2 years who agreed with someone else’s characterization that MIRI was acting against its original mission.
I think lots of individuals at MIRI are intellectually productive and/or high-potential but pretty confused about a lot of things. I don’t currently see a more efficient way to communicate with them than by writing things on the Internet.
I have a long-standing disagreement about AI timelines (I wrote a post saying people are grossly distorting things, which I believe and think is important partially due to the content of my recent posts about my experiences; Anna commented that the post was written in a “triggered” mind state which seems pretty likely given the traumatic events I’ve described). I think lots of people are getting freaked out about the world ending soon and this is wrong and bad for their health. It’s like in Wild Wild Country where the leader becomes increasingly isolated and starts making nearer-term doom predictions while the second-in-command becomes the de-facto social leader (this isn’t an exact analogy and I would be inhibited from making it except that I’m specifically being asked about my political motives, I’m not saying I have a good argument for this).
I still think AI risk is a problem in the long term but I have a broader idea of what “AI alignment research” is, e.g. it includes things that would fall under philosophy/the humanities. I think the problem is really hard and people have to think in inter-disciplinary ways to actually come close to solving it (or to get one of the best achievable partial solutions). I think MIRI is drawing attention to a lot of the difficulties with the problem and that’s good even if I don’t think they can solve it.
Someone I know pointed out that Eliezer’s model might indicate that the AI alignment field has been overall net negative due to it sparking OpenAI and due to MIRI currently having no good plans. If that’s true it seems like a large change in the overall AI safety/x-risk space would be warranted.
My friends and I have been talking with Anna Salamon (head of CFAR) more over this year, she’s been talking about a lot of the problems that have happened historically and how she intends to do different things going forward, and that seems like a good sign but she isn’t past the threshold of willing+able she would need to be to fix the scene herself.
I’m somewhat worried about criticizing these orgs too hard because I want to maintain relations with people in my previous social network, because I don’t actually think they’re especially bad, because my org (mediangroup.org) has previously gotten funding from a re-granting organization whose representative told me that my org is more likely to get funding if I write fewer “accusatory” blog posts (although, I’m not sure if I believe them about this at this time, maybe writing critiques causes people to think I’m more important and fund me more?), because it might spark “retaliation” (which need not be illegal, e.g. maybe people just criticize me a bunch in a way that’s embarrassing, or give me less money). I feel weird criticizing orgs that were as good for my career as they were even though that doesn’t make that much sense from an ethical perspective.
I very much don’t think the central orgs can accomplish their goals if they can’t learn from criticisms. A lot of the time I’m more comfortable in rat-adjacent/postrat-ish/non-rationalist spaces than central rationalist spaces because they are less enamored by the ideology and the central institutions. It’s easier to just attend a party and say lots of weird-but-potentially-revelatory things without getting caught in a bunch of defensiveness related to the history of the scene. One issue with these alternative social settings is that a lot of these people think it’s normal to take ideas less seriously in general so they think e.g. I’m only speaking out about problems because I have a high level of “autism” and it’s too much to expect people to tell the truth when their rent stream depends on them not acknowledging it. I understand how someone could come to this perspective but it seems somewhat of a figure-ground inversion that normalizes parasitic behavior.
I found this a very helpful and useful comment, and resonate with various bits of it (I also think I disagree with a good chunk of it, but a lot of it seems right overall).
I’m curious which parts resonate most with you (I’d ordinarily not ask this because it would seem rude, but I’m in a revealing-political-motives mood and figure the actual amount of pressure is pretty low).
I share the sense that something pretty substantial changed with MIRI in ~2017 and that something important got lost when that happened. I share some of the sense that people’s thinking about timelines is confused, though I do think overall pretty short timelines are justified (though mine are on the longer end of what MIRI people tend to think, though much shorter than yours, IIRC). I think you are saying some important things about the funding landscape, and have been pretty sad about the dynamics here as well, though I think the actual situation is pretty messy and some funders are really quite pro-critique, and some others seem to me to be much more optimizing for something like the brand of the EA-coalition.
I feel like this topic may deserve a top-level post (rather than an N-th level comment here).
EDIT: I specifically meant the “MIRI in ~2017” topic, although I am generally in favor of extracting all other topics from Jessica’s post in a way that would be easier for me to read.
What, if any, are your (major) political motives regarding MIRI/CFAR/similar?
I really liked MIRI/CFAR during 2015-2016 (even though I had lots of criticisms), I think I benefited a lot overall, I think things got bad in 2017 and haven’t recovered. E.g. MIRI has had many fewer good publications since 2017 and for reasons I’ve expressed, I don’t believe their private research is comparably good to their previous public research. (Maybe to some extent I got disillusioned so I’m overestimating how much things changed, I’m not entirely sure how to disentangle.)
As revealed in my posts, I was a “dissident” during 2017 and confusedly/fearfully trying to learn and share critiques, gather people into a splinter group, etc, so there’s somewhat of a legacy of a past conflict affecting the present, although it’s obviously less intense now, especially after I can write about it.
I’ve noticed people trying to “center” everything around MIRI, justifying their actions in terms of “helping MIRI” etc (one LW mod told me and others in 2018 that LessWrong was primarily a recruiting funnel for MIRI, not a rationality promotion website, and someone else who was in the scene 2016-2017 corroborated that this is a common opinion), and I think this is pretty bad since they have no way of checking how useful MIRI’s work is, and there’s a market for lemons (compare EA arguments against donating to even “reputable” organizations like UNICEF). It resembles idol worship and that’s disappointing.
This is corroborated by some other former MIRI employees, e.g. someone who left sometime in the past 2 years who agreed with someone else’s characterization that MIRI was acting against its original mission.
I think lots of individuals at MIRI are intellectually productive and/or high-potential but pretty confused about a lot of things. I don’t currently see a more efficient way to communicate with them than by writing things on the Internet.
I have a long-standing disagreement about AI timelines (I wrote a post saying people are grossly distorting things, which I believe and think is important partially due to the content of my recent posts about my experiences; Anna commented that the post was written in a “triggered” mind state which seems pretty likely given the traumatic events I’ve described). I think lots of people are getting freaked out about the world ending soon and this is wrong and bad for their health. It’s like in Wild Wild Country where the leader becomes increasingly isolated and starts making nearer-term doom predictions while the second-in-command becomes the de-facto social leader (this isn’t an exact analogy and I would be inhibited from making it except that I’m specifically being asked about my political motives, I’m not saying I have a good argument for this).
I still think AI risk is a problem in the long term but I have a broader idea of what “AI alignment research” is, e.g. it includes things that would fall under philosophy/the humanities. I think the problem is really hard and people have to think in inter-disciplinary ways to actually come close to solving it (or to get one of the best achievable partial solutions). I think MIRI is drawing attention to a lot of the difficulties with the problem and that’s good even if I don’t think they can solve it.
Someone I know pointed out that Eliezer’s model might indicate that the AI alignment field has been overall net negative due to it sparking OpenAI and due to MIRI currently having no good plans. If that’s true it seems like a large change in the overall AI safety/x-risk space would be warranted.
My friends and I have been talking with Anna Salamon (head of CFAR) more over this year, she’s been talking about a lot of the problems that have happened historically and how she intends to do different things going forward, and that seems like a good sign but she isn’t past the threshold of willing+able she would need to be to fix the scene herself.
I’m somewhat worried about criticizing these orgs too hard because I want to maintain relations with people in my previous social network, because I don’t actually think they’re especially bad, because my org (mediangroup.org) has previously gotten funding from a re-granting organization whose representative told me that my org is more likely to get funding if I write fewer “accusatory” blog posts (although, I’m not sure if I believe them about this at this time, maybe writing critiques causes people to think I’m more important and fund me more?), because it might spark “retaliation” (which need not be illegal, e.g. maybe people just criticize me a bunch in a way that’s embarrassing, or give me less money). I feel weird criticizing orgs that were as good for my career as they were even though that doesn’t make that much sense from an ethical perspective.
I very much don’t think the central orgs can accomplish their goals if they can’t learn from criticisms. A lot of the time I’m more comfortable in rat-adjacent/postrat-ish/non-rationalist spaces than central rationalist spaces because they are less enamored by the ideology and the central institutions. It’s easier to just attend a party and say lots of weird-but-potentially-revelatory things without getting caught in a bunch of defensiveness related to the history of the scene. One issue with these alternative social settings is that a lot of these people think it’s normal to take ideas less seriously in general so they think e.g. I’m only speaking out about problems because I have a high level of “autism” and it’s too much to expect people to tell the truth when their rent stream depends on them not acknowledging it. I understand how someone could come to this perspective but it seems somewhat of a figure-ground inversion that normalizes parasitic behavior.
I found this a very helpful and useful comment, and resonate with various bits of it (I also think I disagree with a good chunk of it, but a lot of it seems right overall).
I’m curious which parts resonate most with you (I’d ordinarily not ask this because it would seem rude, but I’m in a revealing-political-motives mood and figure the actual amount of pressure is pretty low).
I share the sense that something pretty substantial changed with MIRI in ~2017 and that something important got lost when that happened. I share some of the sense that people’s thinking about timelines is confused, though I do think overall pretty short timelines are justified (though mine are on the longer end of what MIRI people tend to think, though much shorter than yours, IIRC). I think you are saying some important things about the funding landscape, and have been pretty sad about the dynamics here as well, though I think the actual situation is pretty messy and some funders are really quite pro-critique, and some others seem to me to be much more optimizing for something like the brand of the EA-coalition.
I feel like this topic may deserve a top-level post (rather than an N-th level comment here).
EDIT: I specifically meant the “MIRI in ~2017” topic, although I am generally in favor of extracting all other topics from Jessica’s post in a way that would be easier for me to read.
Thanks, this is great (I mean, it clarifies a lot for me).