Slack enables exploration.
Exploration enables exploitation.
Exploitation destroys slack.
Viliam
Karma: 21,666
As I understand it, Russia still perceives itself as a superpower in a decades-long cold war against USA. The fall of Soviet Union was a temporary setback, but now they are back in the game.
From Russian perspective, there are currently only two (or maybe three—I have no idea how Russia perceives China) agents on this planet. Everyone else is an NPC. Some states are “NPCs owned by USA”. Some states are “NPCs owned by Russia”. Other NPCs are neutral and passive. But there are only two (or three, if also China) player characters who have actual agency, and everything that happens on this planet should be interpreted as a military move made by one of them. Any other interpretation means falling for someone’s propaganda, hook, line, and sinker. From Russian perspective, explaining things from any other perspective makes you either a liar (which is a good thing, considering the alternative), or an idiot (if you actually believe what you say).
If you do not grok this perspective, you simply do not understand what Russians actually mean by the things they are saying, e.g. when Putin makes a speech to the Russian public. I am not commenting here on what Putin actually believes—I have no idea about that; I am not a mind reader. I am commenting on the model of reality that his audience has; that most of his audience spent their entire lives surrounded by, so they take it as obvious, as an assumption they do not question.
If you grew up in Eastern Europe, as I did, this perspective is simply “how the pro-Russian people from the older generations see the world”. If you grew up further towards the West, it probably requires an explicit explanation (and probably still remains hard to believe). Which is what I am trying to do here.
(I feel a bit weird about calling it the “Russian perspective”, because some years ago, I would probably strongly argue that it is a “Soviet perspective”, unrelated to any specific nation. And yet, somehow, Soviet Union is gone, but the perspective remains, as strong as ever.)
Ok, so how does this apply to current events? From Russian perspective, there is no fundamental difference between “people in Ukraine overthrow their government and decide to move towards EU and NATO” and “Russia organizes a sham referendum in occupied parts of Ukraine where people vote at gunpoint and the votes are probably not even really counted because the results are already known in advance”. If you say that there is a difference, you are merely parroting American propaganda. The proper response is to say Russian propaganda, where the people of the sovereign Donetsk and Luhansk republics expressed their free will and their deepest desires; as opposed to the Maidan Revolution, which was merely a revolt staged by CIA agents.
Note that the person who is telling you this does not necessarily believe that this is a factual description of reality! From their perspective, they are merely responding to your propaganda by their propaganda, which is what internet was made for. (Yes, they are lying, but so what, you started first!) The factual reality, from Russian perspective, is that Ukraine is an NPC. Talking about what “Ukrainians want” does not correspond to anything in the territory; these are just meaningless soundbites used in propaganda. What actually happened is that Player USA moved some pieces on the chessboard, and then Player Russia moved some pieces on the chessboard. Each side insisting that the pieces “have moved themselves”, which is of course patently absurd to anyone who is not an idiot, but it is a thing the players traditionally say when they are moving the pieces.
I am saying this to explain that if you talk to a person who has adopted the Russian perspective, it is utterly futile to use arguments such as “look, doesn’t it just make sense that many Ukrainians have compared the quality of life in EU countries versus the quality of life in Russia (especially outside its major cities), plus they had some resentment about the Holodomor, so they decided to join the former rather than be gradually reconquered by the latter?”. From their perspective, it makes about as much sense as saying “look, I did not capture your knight in the chess game we played, your knight simply decided to retire; that makes a lot of sense from the perspective of the chess knight, doesn’t it?”. Like, of course, people in Ukraine, or anywhere else, may have their opinions and feelings, but it is naive to assume that this might somehow translate to real-world actions. Ordinary humans are not capable of that! Only the masterminds in Kremlin and Pentagon can actually make things happen, using the ordinary people as their pawns. Even the feelings of the ordinary people are mostly a result of the propaganda they were exposed to, so we might as well disregard them and focus on the primary cause instead.
(Related LW concept: Double-Crux. You can’t change someone’s mind without addressing the underlying different assumptions first. The same statements will be interpreted differently in different perspectives.)
*
Now, to address the article directly:
Putin has a track record of escalating apparently (this needs more data) and Russia seems to be planning for escalation until the war is won.
Please correct me if I am mistaken, but as far as I know, Putin has a track record of escalating against weaker opponents. It is a weak evidence that he might escalate against a stronger opponent (I mean, it certainly is an evidence that he is not a pacifist), but it is not a strong evidence. Fighting against a stronger opponent is qualitatively quite different from fighting against a weaker opponent.
Putin has called for a ceasefire.
Strong connotational objection! Russia promised not to attack Ukraine in 1994 (Budapest memorandum), then attacked anyway, then negotiated a ceasefire in 2014 (Minsk agreements), then attacked again. Considering this history, I don’t see how Putin’s ceasefire could mean anything other than “please give me some time to rebuild my army, then I will attack again”.
In before someone points out the technicalities that Russia uses as an excuse why Budapest memorandum and Minsk agreements do not matter: yes, you have a point! And my point is that whatever agreement would be signed now, Russia would find a similar technicality in the future. Can you honestly believe they would not? (According to the Russian perspective described above, any action against Russia made by anyone is an evidence that Player USA is violating the agreement, which gives Player Russia the right to violate it, too.)
I will put it more strongly: do we have any evidence about Russia being able to live in peace with any of its neighbors, unless said neighbor is backed by the West (i.e. a NATO or EU member) or China (i.e. Mongolia)? Heck, Lukashenka is their puppet, and Putin is talking about annexing Belarus anyway.
Ukraine has “accelerated” its application to join NATO.
As far as I know, only Zelenskyy is talking about “acceleration”. As far as I know, NATO is not taking new members while they are having an active military conflict. I have not heard any NATO representative say that they consider changing that rule.
Ukraine will keep attacking the annexed territories in order to take them back until Russia uses a tactical nuke out of desperation
Objection against “out of desperation”. How is it desperation to lose something that you didn’t own yesterday, just tried to take from someone and failed. (Yes, I am sure that Russia will spin it as desperation, but it is not.)
Conclusion:
Yes, I see a significant risk that Russia will be a sore loser and drop a nuke on Ukraine after losing the war.
(In my model, this happens for a completely different reason. In my model, which of course may be wrong, Russian strategy is simply a precommitment to make any resistance to them as costly as possible, giving their opponents an incentive to give up when attacked. Essentially: “you have two possible futures: either you lose or you win, and we will do everything we can to make sure that the latter option is more painful to you”. This model explains many things that otherwise would not make much sense, for example why Russians torture and execute so many civilians right before they are going to lose some territory. It does not bring them much positive utility, but it decreases the opponent’s utility from winning. Similarly, dropping a nuke on Ukraine right after Ukraine wins would not help them in current situation. But it would send a strong message to countries invaded by Russia in future to think twice about resisting: you lose even if you win!)
Problem is, I am not sure that the alternative is better. The alternative means setting a precedent that whenever Russia makes a suprise attack and organizes a sham referendum the moment they stop gaining territory, everyone must stop fighting and accept that the territory conquered so far is now a part of Russia, forever, under the threat of nuclear holocaust. Consider the incentives this creates for Russia: any attack on any of its non-NATO neighbors has a limited downside and permanent gain. -- The only strategy the neighbors can use against this is to develop their own nukes, or to join NATO as soon as possible. But Russia knows this, so it has an incentive to attack them as soon as possible.
(And then, when Russia finally runs out of non-NATO, non-China-friendly, non-nuke-owning neighbors, what happens next? Eternal peace? Or a nuclear war anyway, with Russia stronger than now?)
We got lucky once with Soviet Union—it collapsed internally, without a nuclear war. Maybe one day we will get similarly lucky with Russian Federation, too. But if not, and if Russia never stops expanding (and so far, I think the evidence points towards Russia keeping expanding), at one moment a conflict will happen. Maybe one nuke dropped now, and then a conventional attack reducing Moscow to a pile of rubble, is the better branch. Or maybe not. Predicting future is difficult, unexpected things may happen, a superintelligent AI may make this all irrelevant soon.
Unfortunately, I do not include “Putin is an old guy, may die any moment from natural causes” to the list of unexpected things that might make the problem go away. Putin is not the problem, it is the organization (called various names, including Cheka, KGB, FSB) that created him, and will create similar Russian leaders in the future.
EDIT:
Another connotational objection: Why the big NATO logo? (Instead of e.g. Putin’s face.)
Jessicata, I will be blunt here. This article you wrote was [EDIT: expletive deleted] misleading. Perhaps you didn’t do it on purpose; perhaps this is what you actually believe. But from my perspective, you are an unreliable narrator.
Your story, original version:
I worked for MIRI/CFAR
I had a psychotic breakdown, and I believed I was super evil
the same thing also happened to a few other people
conclusion: MIRI/CFAR is responsible for all this
Your story, updated version:
I worked for MIRI/CFAR
then Michael Vassar taught me that everyone is super evil, including CFAR/MIRI, and told me to use drugs in order to get a psychotic breakdown and liberate myself from evil
I actually used the drugs
I had a psychotic breakdown, and I believed I was super evil
the same thing also happened to a few other people
conclusion: I still blame MIRI/CFAR, and I am trying to downplay Vassar’s role in this
If you can’t see how these two stories differ, then… I don’t have sufficiently polite words to describe it, so let’s just say that to me these two stories seem very different.
Lest you accuse me of gaslighting, let me remind you that I am not doubting any of the factual statements you made. (I actually tried to collect them here, to separate them from the long stream of dark insinuations.) What I am saying is that you omitted a few “details”, which perhaps seem irrelevant to you, but in my opinion fundamentally change the meaning of the story.
At this moment, we just have to agree to disagree, I guess.
In my opinion, the greatest mistake MIRI/CFAR made in this story, was being associated with Michael Vassar in the first place (and that’s putting it mildly; at some moment it seemed like Eliezer was in love with him, he so couldn’t stop praising his high intelligence… well, I guess he learned that “alignment is more important than intelligence” applies not just to artificial intelligences but also to humans), providing him social approval and easy access to people who then suffered as a consequence. They are no longer making this mistake. Ironically, now it’s you, after having positioned yourself as a victim, who is blinded by his intelligence, and doesn’t see the harm he causes. But the proper way to stop other people from getting hurt is to make it known that listening too much to Vassar does this, predictably. So that he can no longer use the rationalist community as a “social proof” to get people’s trust.
EDIT: To explain my unkind words “after having positioned yourself as a victim”, the thing I am angry about is that you publicly describe your suffering as a way to show people that MIRI/CFAR is evil. But when it turns of that Michael Vassar is more directly responsible for it, suddenly the angle changes and he actually “helped you”.
So could you please make up your mind? Is having a psychotic breakdown and spending a few weeks catatonic in hospital a good thing or a bad thing? Is it trauma, or is it jailbreaking? Because it seems like you call it bad when you attribute it to MIRI/CFAR, but when other people suggest that Vassar was responsible, then it seems a bit like no big deal, definitely not anything to blame him for.
Bring up Genius
Require contributions in advance
Impostor Syndrome as skill/dominance mismatch
Just thinking loudly about the boundaries of this effect...
Suppose I have a garden, and a robot that can pick apples. I instruct the robot to bring me the “nearest big apple” (where “big” is exactly defined as having a diameter at least X). Coincidentally, all apples in my garden are small, so the robot picks the nearest apple at neighbor’s garden and brings it to me, without saying anything.
When the neighbor notices it, he will be angry at me, but I can defend myself that I made an innocent mistake; I didn’t mean to steal apples. This may be a good excuse for the first time; but if the thing keeps happening, the excuse no longer works; it was based on me not knowing.
Also, the robot will not even try to be inconspicuous; it will climb straight over the fence, in the middle of a day, even when the neighbor is there and looking at it. If I try to avoid this, by giving instructions like “bring me the nearest apple, but make sure no one notices you”, I lose the plausible deniability. (I am assuming here that if my neighbor decides to sue me, the judge will see the robot’s programming.)
Now why specifically does the situation change when instead of a robot I use a slave? It seems to me the only difference is that I can (knowingly or not, or anything between that) achieve an outcome where the slave will steal for me, using his intelligence to avoid detection; and if caught anyway, I can afterwards claim that the slave decided to steal on his own will; not because of my commands (either ordering him to steal, or just failing to mention that theft is prohibited), but specifically and intentionally against them. That relieves of all blame.
Okay, so how specifically do I make a slave steal for me, without giving the explicit order? I have to give a command that is impossible (or highly unlikely) to accomplish otherwise. For example, my garden only contains small apples, and I command the slave to bring me a big apple; threatening to whip him if he fails. The only way to avoid whipping is to steal, but hey, I never mentioned that! Also, it is a common knowledge that (unlike the robot) the slave is supposed to know that stealing is wrong.
But this naive strategy fails if the slave checks my garden and then tells me “master, your garden only contains small apples; what am I supposed to do?” Saying “I don’t care, just bring the f-ing apple or feel my wrath” will achieve the purpose, but it puts a dent into my deniability. To do things properly, I must be able to claim afterwards that I believed that my garden contains big apples, and therefore I believed that my orders can be fulfilled legitimately. (Bonus points if afterwards, despite the slave’s testimony of the contrary, I can insist with indignation that my garden still contains big apples, i.e. the slave is just lying to save his ass.)
Therefore, I need to remove the communication channel where the slave could tell me that there are no big apples in my garden. A simple approach is to give the command, along with the threat, and then leave. The slave knows that he gets whipped if the apple is not on my table by sunset, and I am not there to communicate the fact that the order cannot be achieved legitimately. So he either takes the risk and waits for my return—betting on my sense of fairness, that I wouldn’t punish him for not having accomplished the impossible—or decides that it is relatively safer to simply steal the apple. This is better, but still too random. I could increase the probability of stealing by cultivating an image of a hot-tempered master who punishes firsts and asks questions later. But there are still ways this could fail (e.g. the slave could say afterwards “master, this was the last big apple in your garden, the remaining ones are all small”, which would remove my deniability for giving the same command the next day).
A more complex approach is to make it know that talking about small apples is a taboo, and all violations of this taboo will be punished. Now the slave will not dare mention that the apples in my garden are small, but now I need a plausible excuse for the taboo. Is perhaps my garden a source of my pride, and thus I take any criticism of my garden as a personal offense? That could work. I only need to establish the rule “I am proud of my garden and any kind of criticism will be severely punished” sufficiently long before I start ordering my slaves to steal; to avoid the impression that I established that rule exactly for that purpose.
Summary: a command that is impossible (or unlikely) to be accomplished legitimately; a threat of punishment; and proactively destroying the feedback channel under some pretense.
Now I’d like to give some advice on how to notice when you are in a similar situation—where the feedback channels are sabotaged, and perhaps it’s just a question of time when you receive the impossible command, and will have to use your own will to break the rules and take the responsibility for the decision—but actually, all situations with power differential are to smaller or greater degree like this. People usually can’t communicate with their superiors openly. (Not even when the boss says “actually, I prefer if you communicate with me openly”. Seriously, don’t. What that statement actually means is that talking about difficulties in communication to superiors is also a taboo. Generally, a company statement “X is true” is usually best translated as “saying ‘X is false’ will be punished”.)
A practical advice would perhaps be to notice the local taboos, and maneuver to avoid coming into contact with them. If a person X cannot be criticized, avoid being on the same team as X. If software Y cannot be criticized, avoid work that involves using software Y. If ideology Z cannot be criticized, avoid projects where Z applies most strongly. Yeah, I know, this is easier said than done, and it does not work reliably (e.g. you can choose a different team, and the next day X switches to that team, too).
RationalWiki on face masks
I think you get it mostly right, and then you just make a different conclusion.
The part where you agree is:
We do not have a scientific understanding of how to tell a superintelligent machine to [solve problem X, without doing something horrible as a side effect], because we cannot describe mathematically what “something horrible” actually means to us...
And the conclusion that AI safety people make is:
...and that is a problem, because in the following years, machines smarter than humans are likely to come, and they may do things with horrible side effects that their human operators will not predict.
While your conclusion seems to be:
...therefore people should be ashamed for talking about this topic.
So, if you want to be a proper Popperian, you probably need to sit and wait until actual superintelligent machines are made and actually start doing horrible things, and then (assuming that you survive) you can collect and analyze examples of the horrible things happening, propose falsifiable hypotheses on how to avoid these specific horrible things happening again, do the proper experiments, measure the p-values, and publish in respected scientific journals. This is how respectable people would approach the problem.
The alternative is to do the parts that you can do now… and handwave the rest of it, hoping that later someone else will fill in the missing parts. For example, you can collect examples of surprising things that current (not superintelligent) machines are making when solving problems. And the handwavy part is ”...and now imagine this, but extrapolated for a superintelligence”.
Or you can make a guess about which mathematical problems may turn out to be relevant for AI safety (although you cannot be sure you guessed right), and then work on those mathematical problems rigorously. In which case the situation is like: “yeah, this math problem is solved okay from the scientific perspective, it’s just its relevance for AI safety that is dubious”.
I am not familiar with the AI safety research, so I cannot provide more information about it. But my impression is that it is similar to a combination of what I just described: examples of potential problems (with non-superintelligent machines), and mathematical details which may or may not be relevant.
The problem with “pop Popperianism” is that it describes what to do when you already have a scientific hypothesis fully formed. It does not concern itself with how to get to that point. Yes, the field of AI safety is currently mostly trying to get to that point. That is the inevitable first step.
Please allow me to point out one difference between the Rationalist community and Leverage that is so obvious and huge that many people possibly have missed it.
The Rationalist community has a website called LessWrong, where people critical of the community can publicly voice their complaints and discuss them. For example, you can write an article accusing their key organizations of being abusive, and it will get upvoted and displayed on the front page, so that everyone can add their part of the story. The worst thing the high-status members of the community will do to you is publicly post their disagreement in a comment. In turn, you can disagree with them; and you will probably get upvoted, too.
Leverage Research makes you sign an NDA, preventing you from talking about your experience there. Most Leverage ex-members are in fact afraid to discuss their experience. Leverage even tries (unsuccessfully) to suppress the discussion of Leverage on LessWrong.
Considering this, do you find it credible that the dynamics of both groups is actually very similar? Because that seems to be the narrative of the post we are discussing here—the very post that got upvoted and is displayed publicly to insiders and outsiders alike. I do strongly object against making this kind of false equivalence.
it’s pretty important to be willing to engage the hidden / silent / unpopular minority
The hidden / silent / unpopular minority members can post their criticism of MIRI/CFAR right here, and most likely it will get upvoted. No legal threats whatsoever. No debugging sessions with their supervisor. Yes, some people will probably disagree with them, and those will get upvoted, too.
You know, this reminds me of comparison between dictatorships and democracies. In a dictatorship, the leader officially has a 100% popular support. In a democracy, maybe 50% of people say that the country sucks and the leadership is corrupt. Should we take these numbers at the face value? Should we even discount them both to the same degree and say “if the dictatorship claims to have 100% popular support, but in fact only 20% of people are happy with the situation, then if the democracy claims to have 50% popular support, we should apply the same ratio and conclude that only 10% of people are happy?”.
Because it seems to me that you are making a similar claim here. We know that some people are afraid to talk publicly about their experience in Leverage. You seem to assume that there must be a similar group of people afraid to talk publicly about their experience in MIRI/CFAR. I think this is unlikely. I assume that if someone is unhappy about MIRI/CFAR doing something, there is probably a blog post about it somewhere (not necessarily on LessWrong) already.
On the other side of it, why do people seem TOO DETERMINED to turn [Michael Vassar] into a scapegoat?
Do you disagree with specific actions being attributed to Michael? Do you disagree with the conclusion that it is a good reason to avoid him and also tell all your friends to avoid him?
Yeah, it’s nice when your opponents volunteer to remove from you the burden of proof whether they are irrational.
But seriously, I don’t even know where to start. Perhaps here: Articles written on most popular websites are clickbait. It means that their primary purpose is to make you read the article after seeing the headline, and then share it either because you love it or because you hate it. And that’s what you did. Mission accomplished.
Another article on the same website explains why animal rights movements are oppresive. (I am not going to link it, but here are the arguments for the curious readers: because it’s wrong to care about animals while there are more important causes on this planet such as people being oppressed; because vegans and vegetarians don’t acknowledge that vegan or vegetarian food can be expensive; because describing animals as male and female marginalizes trans people; and because protecting animals is colonialistic against native people who hunt animals as part of their tradition.) Obviously, the whole article is an exercise in making the reader scream and share the article to show other readers how crazy it is. This is exactly what the authors and editors get paid for; this is how you shovel the sweet AdSense money on them. So the only winning move is not to play this game.
.
I may be too extreme in this aspect, but when I talk with most people, I simply assume that almost everything they say is a metaphor for something (usually for their feelings), and almost nothing is to be taken literally. This is a normal way of communication among people who couldn’t program a Friendly AI if their very lives depended on it.
When someone says “rationality is bad”, the correct translation is probably something like “I hate my father because he criticized me a lot and didn’t play with me; and my father believes he is smart, and he makes smartness his applause light; and this is why I hate everything that sounds like smartness”. You cannot argue against that. (If you try anyway, the person will not remember any specific thing you said, they will only remember that you are just as horrible person as their father.) This is how people talk. This is how people think. And they understand each other, so when another person who also hates their father hears it, they will get the message, and say something like “yeah, exactly like you said, rationality is stupid”. And then they know they can trust each other on the emotional level.
Here is a short dictionary containing the idioms from the article:
everydayfeminism.com = “I hate my father”
we should abolish prisons, police = “I hate my father”
cisheteropatriarchy = “I hate my father; but I also blame my mother for staying with him”
those who are committed to social justice = “my friends, who also hate their fathers”
we have to stop placing limits on ourselves = “we should steal some money and get high”
Being Rational Has No Inherent Value = “I don’t even respect my father”
my very existence is irrational = “my father disapproves of my lifestyle”
The only logical time for abolition and decolonization is now = “I wish I had the courage to tell my parents right now how much I am angry at them”
You are overanalyzing it, searching for a logical structure when there is none. If you treat the article as a free-form poem, you will get much closer to the essence. You don’t share the author’s emotions, that’s why the text rubs you the wrong way.
And by the way, other political groups do similar things, just in a different flavor (and perhaps intensity).
Some context, please. Imagine the following scenario:
Victim A: “I was hurt by X.”
Victim B: “I was hurt by Y.”
There is absolutely nothing wrong with this, whether it happens the same day, the next day, or week later. Maybe victim B was encouraged by (reactions to) victim A’s message, maybe it was just a coincidence. Nothing wrong with that either.
Another scenario:
Victim A: “I was hurt by X.”
Victim B: “I was also hurt by X (in a different way, on another day etc.).”
This is a good thing to happen; more evidence, encouragement for further victims to come out.
But this post is different in a few important ways. First, Jessicata piggybacks on Zoe’s story a lot, insinuating analogies, but providing very little actual data. (If you rewrote the article to avoid referring to Zoe, it would be 10 times shorter.) Second, Jessicata repeatedly makes comparison between Zoe’s experience at Leverage and her experience at MIRI/CFAR, and usually concludes that Leverage was less bad (for reasons that are weird to me, such as because their abuse was legible, or because they provided space for people to talk about demons and exorcise them). Here are some quotes:
I want to disagree with a frame that says that the main thing that’s bad was that Leverage (or MIRI/CFAR) was a “cult”. This makes it seem like what happened at Leverage is much worse than what could happen at a normal company. But, having read Moral Mazes and talked to people with normal corporate experience (especially in management), I find that “normal” corporations are often quite harmful to the psychological health of their employees, e.g. causing them to have complex PTSD symptoms, to see the world in zero-sum terms more often, and to have more preferences for things to be incoherent.
Leverage was an especially legible organization, with a relatively clear interior/exterior distinction, while CFAR was less legible, having a set of events that different people were invited to, and many conversations including people not part of the organization. Hence, it is easier to attribute organizational responsibility at Leverage than around MIRI/CFAR.
Unless there were psychiatric institutionalizations or jail time resulting from the Leverage psychosis, I infer that Leverage overall handled their metaphysical weirdness better than the MIRI/CFAR adjacent community. While in Leverage the possibility of subtle psychological influence between people was discussed relatively openly, around MIRI/CFAR it was discussed covertly, with people being told they were crazy for believing it might be possible.
Leverage definitely had large problems with these discussions, and perhaps tried to reach more intersubjective agreement about them than was plausible (leading to over-reification, as Zoe points out), but they seem less severe than the problems resulting from refusing to have them, such as psychiatric hospitalization and jail time.
Despite the witch hunts and so on, the Leverage environment seems more supportive than what I had access to. The people at Leverage I talk to, who have had some of these unusual experiences, often have a highly exploratory attitude to the subtle mental realm, having gained access to a new cognitive domain through the experience, even if it was traumatizing.
An ex-Leverage person I know comments that “one of the things I give Geoff the most credit for is actually ending the group when he realized he had gotten in over his head. That still left people hurt and shocked, but did actually stop a lot of the compounding harm.” (While Geoff is still working on a project called “Leverage”, the initial “Leverage 1.0” ended with most of the people leaving.) This is to some degree happening with MIRI and CFAR, with a change in the narrative about the organizations and their plans, although the details are currently less legible than with Leverage.
I hope that those that think this is “not that bad” (perhaps due to knowing object-level specifics around MIRI/CFAR justifying these decisions) consider how they would find out whether the situation with Leverage was “not that bad”, in comparison, given the similarity of the phenomena observed in both cases; such an investigation may involve learning object-level specifics about what happened at Leverage. I hope that people don’t scapegoat; in an environment where certain actions are knowingly being taken by multiple parties, singling out certain parties has negative effects on people’s willingness to speak without actually producing any justice.
...uhm, does this sound a bit like a defense of Leverage, or at least saying “Zoe, your experience in Leverage was not as bad as my experience in MIRI/CFAR”? That is poor taste, especially when the debate about Zoe’s experience hasn’t finished yet.
Third, this comparison and downplaying is made even worse by the fact that many supposed analogies are not that much analogical:
Zoe had mental trauma after her experience in Leverage. Jessicata had mental trauma after her experience in MIRI/CFAR, and after she started experimenting with drugs, inspired by critics of MIRI/CFAR.
Zoe had to sign an NDA, covering lot of what was happening in Leverage, and now she worries about possible legal consequences of her talking about her abuse. Jessicata didn’t have to sign anything… but hey, she was once discouraged from writing a blog on AI timeline… which is just as bad, except much worse because MIRI/CFAR is less transparent about being evil. (Sorry, I am too sarcastic here, I find it difficult to say these things with a straight face.)
Zoe was convinced by Leverage that everything that happened to her was her own fault. Jessicata joined a group of MIRI/CFAR haters who believed that everything was evil but especially MIRI/CFAR, and then she ended up believing that she was evil… yeah, again, fair analogy! Leverage at least tells you openly that you are a loser, but the insidious MIRI/CFAR uses some super complicated plot, manipulating their haters to convince you about the same thing.
etc. (I am out of time, and also being sarcastic is against the norms of LW, so I better end here.)
In summary, it is the combination of: piggybacking on another victim’s story, making analogies that are not really analogies, and then downplaying the first victim’s experience… plus the timing right in the middle of debating the first victim’s experience… that makes it so bad.
I generally worry about all kinds of potential bad actors associating themselves with EA/rationalists.
There seems to be a general pattern where new people come to an EA/LW/ACX/whatever meetup or seminar, trusting the community, and there they meet someone who abuses this trust and tries to extract free work / recruit them for their org / abuse them sexually, and the new person trusts them as representatives of the EA/rationalist community (they can easily pretend to be), while the actual representatives of EA/rationalist community probably don’t even notice that this happens, or maybe feel like it’s not their job to go reminding everyone “hey, don’t blindly trust everyone you meet here”.
I assume the illusion of transparency plays a big role here, where the existing members generally know who is important and who is a nobody, who plays a role in the movement and who is just hanging out there, what kind of behavior is approved and what kind is not… but the new member has no idea about anything, and may assume that if someone acts high-status then the person actually is high-status in the movement, and that whatever such person does has an approval of the community.
To put it bluntly, EA/rationalist community kinda selects for people who are easy to abuse in some ways. Willing to donate, willing to work to improve the world, willing to consider weird ideas seriously—from the perspective of a potential abuser, this is ripe fruit ready to be taken, it is even obvious what sales pitch you should use on them.
Not sure what exactly to do about this, but perhaps the first step could be to write some warnings about this, and read them publicly at the beginning of every public event where new people come. Preferably with specific examples of things that happened in the past; like, not the exact name and place, but the pattern, like “hey, I have a startup that aims to improve the world, wanna code for me this app for free, I will totally donate something to some effective charity, pinky swear”.
How to be skeptical about meditation/Buddhism
I have zero experience in hiring people; I only know how things seem from the opposite side (as a software developer). The MIRI job announcement seems optimized to scare people like me away. It involves a huge sunk cost in time—so you only take it if you are 100% sure that you want the job and you don’t mind the (quite real) possibility of paying the cost and being rejected at the end of the process anyway. If that is how MIRI wants to filter their applicant pool, great. Otherwise, they seem utterly incompetent in this aspect.
For starters, you might look at Stack Exchange and check how potential employees feel about being given a “homework” at an interview. Your process is worse than that. Because, if you instead invited hundred software developers on a job interview, gave them extensive homework, and then every single one of them ghosted you… at least you could get a suspicion that you were doing something wrong, and by experimenting you could find out what exactly it was. If you instead announce the homework up front, sure it is a honest thing to do, but then no one ever calls you, and you think “gee, no one is interested in working on AI alignment, I wonder why”.
How to do it instead?
Here, my confidence is much lower, but it seems to me that you basically need to choose your strategy: do you have specific experts in mind, or are you just hoping to discover some talent currently unknown to you?
If you have specific experts in mind, you need to approach them actively. Not just post an announcement on your website, hope that they notice it, hope that they realize that you meant them, hope that they decide to jump through all the hoops, so that at the end you can tell them: “indeed, I had you in mind; I am glad you got the hints right—now we can discuss salary and work conditions”.
If you are looking for an unknown talent, you probably want some math and/or programming skills, so why not start with some mathematical or programming test, which would filter out the people without the skills. First round, publish the problems on LessWrong and let people type their answers into Google Forms. Yes, cheating is possible. That is why the second round will be done on Zoom, but only with those who passed the first round. Do not be too harsh at grading, just filter out the people who obviously do not have the talent. What is left is the people who have some talent in general, and are kinda interested in the job. Now you can start giving them more specific tasks and see how well they do. Maybe reward them by internet points first, so they can see how they compare to the competition. When the tasks get more difficult, start paying them real money, so that when it becomes too much of a time sink, it also becomes a side income. (Don’t: “If you do this successfully, I will give you X money.” Do: “Try your best, and I will give you X money for trying, as long as you give me something. Of course if the results are not up to our standards, we may at some moment decide to remove you from the list of participants.”) At the end, either everyone fails, or you will have found your person. Even if everyone fails, you still get some data, for example at which step most of them failed.
Yes, there will be some wasted time and effort. It happens to all companies. Whenever you invite someone for an interview, and after an hour you decide that you don’t want them for whatever reason, you just wasted one hour of salary of everyone who participated in that interview. That is a cost of hiring people.
Importantly: you need to have one person whose task is to do the recruitment. If you just expect it to happen, but it’s actually no one’s high priority, then… most likely it won’t.
At the beginning, LessWrong was strongly atheistic:
A Parable on Obsolete Ideologies—goes full Godwin on religion
Later, writing about religion felt like beating a dead horse.
Then, I suppose as a part of the “meta-rationality” wave, the idea of cultural evolution became popular: how people can do the right things for wrong reasons (and premature rationality can hurt you), for example how divination is useful, not because supernatural things are actually true, but because divination is effectively a source of randomness, so it is useful where a random-number generator would be useful, e.g. when fighting an intelligent adversary or hunting animals. This made religion somewhat acceptable—perhaps not epistemically, but instrumentally.
We also had the period when LessWrong was effectively dead and replaced by Slate Star Codex as a center of rationalist discource, but SSC was not explicitly atheist, and at least half of its comment section did not even aspire to be rationalist.
Lately Buddhism became popular on LessWrong—a fact that I hate, and I have already complained about it many times—first it started as “there seems to be some scientific evidence in favor of meditation giving its users some benefits” which is okay if true, but of course the entire expert literature on this topic is full of memetic hazards, and we gradually move towards accepting parts of Buddhist epistemology, or at least privileging them as hypotheses. (Sometimes I imagine a parallel universe where e.g. Catholics have invented push-ups, and the rationalists in that universe progress from “hey, there is a scientific evidence that push-ups are good for your body” towards accepting the wisdom of Catholicism and praying to Lord Jesus.) I sincerely hope that this all is just temporary and we will grow out of it, better sooner than later.
(And I suspect the recent silence on the topic also reflects the recent changes in American politics. To put it bluntly, a decade or more ago, it was like “religion = Republicans = the bad guys”, but then people realized that Islam is a religion too, and being a too consistent atheist also makes you “islamophobic”, which is a politically incorrect thing, and therefore it is better to just avoid this topic.)
I am confused, and I do not have enough energy to figure this out. So I will just wait and see what happens.
After Ben’s post, everything seemed obvious. After the response from Nonlinear, everything seems obvious again. I wonder if it is possible that a third post (and fourth etc.) will flip the consensus again.
I wish that someone who only cares about technical details would go through the claims made in the original article, and made a list of: what exactly was claimed, what exactly was refuted, and how are these two related. To distinguish “X was clearly refuted by evidence Y” from “it is quite possible for X and Y to be simultaneously true” and maybe some other options.
(EDIT: This comment does it well; I would like to see Ben’s response specifically to it.)
I have this weird feeling (and feel weird about admitting it) that the response from Nonlinear sometimes felt too good. Not sure if I can explain it. It’s like… if one guy says “they paid me $1000” and the other guy says “no, we paid him $2000″, then my brain will generate possibilities like: maybe it was $1000, maybe it was $2000, and maybe it was something in between and both sides are exaggerating—all three options seem plausible. But if instead one guy says “they paid me $1000” and the other guy says “actually, we gave him control over our entire budget of $1,000,000 and told him ‘dude, take as much as you like, even all if you want, we don’t mind’”, then my brain just refuses to generate possibilities and assign probabilities, and just displays an error message. (And yes, I am aware that this is exactly the kind of bullshit story someone who simply refuses to update on clear evidence would make up. It is also how a genuine “something is wrong but I can’t figure out what” feels like.)
I was never a strong believer. There was never a moment where my “faith shattered”, because I never had “faith” in the first place. It’s just, given the filtered information, how the regime described the situation, that seemed to me like a plausible description of reality. I haven’t heard any alternative description, and I didn’t have a reason to invent one.
Also, I was a small kid, so my ability to think about politics was quite limited. For example, I heard the broadcast of Voice of America / Radio Free Europe (I am not sure which one, maybe both) a few times, and I was warned that this is something controversial that I am never supposed to mention to anyone outside my family, lest I want to get my family into big trouble… but frankly, I didn’t understand what the broadcast was about. It was just some boring adult talk; I had no idea what was supposed to be exciting.
From my perspective, living in the regime was just an ordinary everyday experience. Like, you are told to go to school, so you go to school, because everyone does, duh. (Imagine a society that never invented the concept of homeschooling.) Then you are told to join the Pioneer movement, so you do, because everyone does, duh. (There are a few exceptions; you are told not to ask. They are problematic people. You don’t want to be problematic, do you? So just do your thing and ignore them.) The Pioneer movement is boring, almost the same way the school is boring. You don’t know why adults organize your life in such a boring way, but that’s how it is, so you sit and listen when you are told to sit and listen, then you go home and play. This was the depth of my political sophistication back then.
When I was 13, the regime fell.
Afterwards, I was exposed to different kinds of information; different people saying contradictory opinions. I guess I don’t have a strong psychological need for closure. I accepted that when they tell me about their near-mode experience, they are probably telling the truth; and when they tell me their far-mode opinions, well, everyone says something different, and there is no way to reconcile it. (Today I would say: everyone focuses on some aspects of reality, and ignores whatever doesn’t fit in their story.)
So my model updated gradually. I gradually collected more and more data points about what was wrong about the regime, starting with “the Pioneer movement meetings are so boring”, through “seems like some people were treated unfairly by the regime, for reasons that seem stupid to me”, through “holy shit, they were actually Hitler’s allies during the first part of WW2, and they murdered millions of people”, to “what the fuck, it’s even worse than I imagined when I already believed it was quite evil”. There was no clear moment when I switched from “pro” to “anti”; it was a gradual shift.
I also don’t know what “social metaphysics” means.
I get the mood of the story. If you look at specific accusations, here is what I found, maybe I overlooked something:
This is like 5-10% of the text. A curious thing is that it is actually the remaining 90-95% of the text that evoke bad feelings in the reader; at least in my case.
To compare, when I was reading Zoe’s article, I was shocked by the described facts. When I was reading Jessica’s article, I was shocked by the horrible things that happened to her, but the facts felt… most of them boring… the most worrying part was about a group of people who decided that CFAR was evil, spent some time blogging against CFAR, then some of them killed themselves; which is very sad, but I fail to see how exactly CFAR is responsible for this, when it seems like the anti-CFAR group actually escalated the underlying problems to the point of suicide. (This reminds me of XiXiDu describing how fighting against MIRI causes him health problems; I feel bad about him having the problems, but I am not sure what MIRI could possibly do to stop this.)
Jessica’s narrative is that MIRI/CFAR is just like Leverage, except less transparent. Yet when she mentions specific details, it often goes somewhat like this: “Zoe mentioned that Leverage did X. CFAR does not do X, but I feel terrible anyway, so it is similar. Here is something vaguely analogical.” Like, how can you conclude that not doing something bad is even worse than doing it, because it is less transparent?! Of course it is less transparent if it, you know, actually does not exist.
Or maybe I’m tired and failing at reading comprehension. I wish someone would rewrite the article, to focus on the specific accusations against MIRI/CFAR, and remove all those analogies-except-not-really with Zoe; just make it a standalone list of specific accusations. Then let’s discuss that.