AI alignment researcher supported by MIRI and LTFF. Working on the learning-theoretic agenda.
Vanessa Kosoy(Vanessa Kosoy)
Karma: 4,147
If you’re asking why I believe that they don’t require presence, I’ve been interviewing with them and that’s my understanding from talking with them.
Are you talking about “you can work from home and come to the office occasionally”, or “you can live on a different continent”?
Sounds pretty much like a safety org to me.
I found no mention of existential risk on their web page. They seem to be a commercial company, aiming at short-to-mid-term applications. I doubt they have any intention to do e.g. purely theoretical research, especially if it has no applications to modern systems. So, what they do can still be meritorious and relevant to reducing existential risk. But, the context of this discussion is: can we replace all AI safety orgs by just one org. And, Anthropic is too specialized to serve such a role.
AFAICT, Anthropic is not an existential AI safety org per se, they’re just doing a very particular type of research which might help with existential safety. But also, why do you think they don’t require physical presence?
I met Vassar once. He came across as extremely charismatic (with a sort of charisma that probably only works on a particular type of people, which includes me), creating the impression of saying wise and insightful things (especially if you lack relevant domain knowledge), while in truth he was saying a lot of stuff which was patently absurd. Something about his delivery was so captivating, that it took me a while to “shake off the fairy dust” and realize just how silly some of his claims were, even when it should have been obvious from the start. Moreover, his worldview seemed heavily based on paranoidal / conspiracy-theory type of thinking. So, yes, I’m not too surprised by Scott’s revelations about him.
This might be the right approach, but notice that no existing AI risk org does that. They all require physical presence.
A “secondary concern” in the sense that, we should work remotely? Or in the sense that everyone should relocate? Because the latter is unrealistic: people have families, friends, communities, not anyone can uproot themself.
I worked for 16 years in the industry, including management positions, including (briefly) having my own startup. I talked to many, many people who worked in many companies, including people who had their own startups and some with successful exits.
The industry is certainly not a rose garden. I encountered people who were selfish, unscrupulous, megalomaniac or just foolish. I’ve seen lies, manipulation, intrigue and plain incompetence. But, I also encountered people who were honest, idealistic, hardworking and talented. I’ve seen teams trying their best to build something actually useful for some corner of the world. And, it’s pretty hard to avoid reality checks when you need to deliver a real product for real customers (although some companies do manage to just get more and more investments without delivering anything until the eventual crash).
I honestly think most of them are not nearly as bad as Leverage.
if one believed somebody else were just as capable of causing AI to be Friendly, clearly one should join their project instead of starting one’s own.
Nitpicking: there are reasons to have multiple projects, for example it’s convenient to be in the same geographic location but not anyone can relocate to any place.
I’m not sure whether you’re talking about hiring in most organizations or hiring in MIRI in particular?
everything I knew about how to be hired would point towards having little mental resistance to organizational narratives
Can you elaborate a little on this?
Full disclosure: I am a MIRI Research Associate. This means that I receive funding from MIRI, but I am not a MIRI employee and I am not privy to its internal operation or secrets.
First of all, I am really sorry you had these horrible experiences.
A few thoughts:
Thought 1: I am not convinced the analogy between Leverage and MIRI/CFAR holds up to scrutiny. I think that Geoff Anders is most likely a bad actor, whereas MIRI/CFAR leadership is probably acting in good faith. There seems to be significantly more evidence of bad faith in Zoe’s account than in Jessica’s account, and the conclusion is reinforced by adding evidence from other accounts. In addition, MIRI definitely produced some valuable public research whereas I doubt the same can be said of Leverage, although I haven’t been following Leverage so I am not confident about the latter (ofc it’s in principle possible for a deeply unhealthy organization to produce some good outputs, and good outputs certainly don’t excuse abuse of personnel, but I do think good outputs provide some evidence against such abuse).
It is important not to commit the fallacy of gray: it would risk both judging MIRI/CFAR too harshly and judging Leverage insufficiently harshly. The comparison Jessica makes to “normal corporations” reinforces this impression: I have much experience in the industry, and although it’s possible I’ve been lucky in some ways, I still very much doubt the typical company is nearly as bad as Leverage.
Thought 2: From my experience, AI alignment is a domain of research that intrinsically comes with mental health hazards. First, the possibility of impending doom and the heavy sense of responsibility are sources of stress. Second, research inquiries often enough lead to “weird” metaphysical questions that risk overturning the (justified or unjustified) assumptions we implicitly hold to maintain a sense of safety in life. I think it might be the closest thing in real life to the Lovecraftian notion of “things that are best not to know because they will drive you mad”. Third, the sort of people drawn to the area and/or having the necessary talents seem to often also come with mental health issues (I am including myself in this group).
This might be regarded as an argument to blame MIRI less for the mental health fallout described by Jessica, but this is also an argument to pay more attention to the problem. It would be best if we could provide the people working in the area with the tools and environment to deal with these risks.
Thought 3: The part that concerned me the most in Jessica’s account (in part due to its novelty to me) is MIRI’s internal secrecy policy. While it might be justifiable to have some secrets to which only some employees are privy, it seems very extreme to require going through an executive because even the mere fact that a secret project exists is too dangerous. MIRI’s secrecy policy seemed questionable to me even before, but this new spin makes it even more dubious.
Overall, I wish MIRI was more transparent, so that for example its supporters would know about this internal policy. I realize there are tradeoffs involved, but I am not convinced MIRI chose the right balance. To me it feels like overconfidence about MIRI’s ability to steer the right way without the help of external critique.
Moreover, I’m a little worried that MIRI’s lack of transparency might pose a risk for the entire AI safety project. Tbh, one of my first thoughts when I saw the headline of the OP was “oh no, what if some scandal around MIRI blows up and the shockwave buries the entire community”. And I guess some people might think this is a reason for more secrecy. IMO it’s a reason for less secrecy (not necessarily less secrecy about technical AI stuff, but less secrecy about management and high-level plans). If we don’t have any skeletons in the closest, we don’t need to worry about the day they will come out. And eventually everything comes out, more or less. When most of everything is in the open, the community can find the right balance around it, and the reputation system is much more robust.
Thought 4: “Someone in the community told me that for me to think AGI probably won’t be developed soon, I must think I’m better at meta-rationality than Eliezer Yudkowsky, a massive claim of my own specialness.” I think (hope?) this is not at all a prevalent stance in the community (or at least in its leading echelons), but just for the record I want to note my strong position that the “someone” in this story is very misguided. Like I said, I don’t think community is currently comparable to Leverage, but this is the sort of thing that can push us in that direction.
He’s part of a small club of people that have led more than one country, and might be the only one who did so simultaneously.
That’s not really true, personal unions were quite common historically.
Personally I think this story is an important warning about how people with a LW-adjacent mindset can death spiral off the deep end. This is something that happened around this community multiple times, not just in Leverage (I know of at least one other prominent example and suspect there are more), so we should definitely watch out for this and/or think how to prevent this kind of thing.
The nominal GDP is given in units of currency, but the value of currency can change over time. Today’s dollars are not the same as the dollars of 1900. When I wrote the previous comment, I thought that’s handled using a consumer price index, in which case the answer can depend on which goods you include in the basket. However, actually real GDP is defined using something called the GDP deflator which is apparently based on a variable “basket” consisting of those goods that are actually traded, in proportion to the total market value traded in each one.
AFAIU, this means GDP growth can theoretically be completely divorced from actual value. For example, imagine there are two goods, A and B, s.t. during some periods A is fashionable and its price is double the price of B, whereas during other periods B is fashionable and its price is double the price of A. Assume also that every time a good becomes fashionable, the entire market switches to producing almost solely this good. Then, every time the fashion changes the GDP doubles. It thus continues to grow exponentially while the real changes are just circling periodically on the same place. (Let someone who understands economics correct me if I misunderstood something.)
Given the above, we certainly cannot rule out indefinite exponential GDP growth. However, I think that the OP’s argument that we live in a very unusual situation can be salvaged by using a different metric. For example, we can measure the entropy per unit of time produced by the sum total of human activity. I suspect that for the history so far, it tracks GDP growth relatively well (i.e. very slow growth for most of history, relatively rapid exponential growth in modern times). Since the observable universe has finite entropy (due to the holographic principle), there is a bound on how long this phenomenon can last.
Soonbegon didn’t “survive to equilibrium” despite being called a winner, so I probably shouldn’t count it.
I think it did survive in the Benthic?
In the temperate forest, the Snark is listed as extinct in generation 140 but then it appears in the table of winners. Does it mean it was reintroduced from other biomes?
Well, the Snark also survived in the Human Garbage Dump. [EDIT: And in the Temperate Forest and the Rainforest]
What do you mean, you just started it in the Temperate Forest? The Snark comes from the Temperate Forest.
Now it’s the time to admit I created species without any thought for strategy, but solely by the principle of thinking of cool names and then assigning attributes which seemed suitable (yes, it did lead to me to submit a bunch of species with weapons + armor > 10). The fact one of my own survived anyway confirms what I suspected, namely that the game is so complicated that few people could find reliably successful strategies (or maybe it’s just the random?) I guess it also helped that I submitted 10 species and most players submitted much less.
As usual, the incentive for voicing disagreement is higher than the incentive for voicing agreement, so to balance that I just want to say that personally I liked the rituals (but I’m also open to experimenting with other formats).
On a different note, how do you know how many people opened the email?