The summary judgement doesn’t cite Dean in particular but does cite the amicus brief and number of signers twice (search for “176”). Seems like Jeff Dean’s name may not matter much but raising the # of signers by 30% might have?
Elizabeth
Karma: 23,451
It seems like you’re assuming that work labeled as safety is safety, or at least more safety than capabilities.
In the ancestral environment, infection was a top killer. Today it’s a rounding error. Our immune systems are therefore set by evolution to be far too sensitive for the modern environment.
Definitely for certain kinds of diseases (macroparasites, anything where humans aren’t the primary resevoir), maybe for others (septicemia? things spread by water or mosquito?) but e.g. respiratory illnesses are much more abundant now.
There exist watermarking schemes such that it is completely impossible for users to ever distinguish between watermarked and non-watermarked text. (A tempting but invalid argument is that no such scheme can exist because watermarking consumes entropy.)
I agree that this is the crux, but have yet to see anyone making this claim provide evidence or an explanation as to how it is possible.
People like brevity but will punish authors for leaving things out so authors defend themselves with length.
Because some people need to set “Reign of Terror” in order to feel comfortable doing any moderation at all.
Additionally, Wei Dai had not set Reign of Terror when the comments were made.
Why can’t we have nice things? Like, specifically?
See the ensuing conversation with Habryka. I assumed this because people claimed it, and MIRI’s defenders responded with “this does not meet the technical definition of paying off aggressors”, and not the straightforward “MIRI never said they wouldn’t pay off aggressors”. Sounds like I was wrong, but to this day that response only makes sense to me if the writer is conceding that MIRI committed to never paying blackmail.
I think it would help to fully explain the chain of thought risk in this post, even if it’s redundant with the link. Most people don’t click links, and it’s an important argument.
The big take home from @TurnTrout’s post on his lobbying efforts within google is that they didn’t work, Google signed an “all lawful use” contract with the pentagon. But he did (reportedly) get Jeff Dean to sign an amicus brief in Anthropic’s lawsuit agains the pentagon. Does anyone know what the value of that is? If it gets Anthropic marginally more damages, or more protection from a designation as a supply chain risk, that seems value-creating.
Pretty much anyone in academia will tell prospective students “don’t do a phd” or at least “you should only do a phd if narrow criteria are met
My undergrad professors heavily encouraged me to reapply to grad school (in biology) because I was so special and good.
In hindsight, given the poor track record of large alignment-branded research organizations, it was a significant mistake for me to accept Resolution’s offer without having thought much more carefully about ways that joining might be bad (such as the points above). There’s a common failure mode whereby rationalists and EAs are averse to thinking about how their cognition is skewed by incentives and power dynamics; though I often talk about this issue abstractly, I still failed to account for it in this case.
I’d be very interested in a 5-whys on this.
Leverage was also much more lonely than love-bomb-ey
cutting off relationships with people outside the cult
This is a terrible sign except for the times it’s extremely helpful because the people being cut off were terrible. Queer culture supports/encourages people to cut off hateful, homophobic family. Recovery programs encourage people to cut off their user friends. Whether this is harmful depends on the quality of the people being cut off.
I’m interested in why the discourse made discussion impossible
The vitriol against the leader made people feel socially unsafe to speak out about the good things they’d gotten from him. Mostly they feared the vitriol would be turned on them for “defending” the designated bad guy (who TBC generally was pretty bad, but if they were all bad people wouldn’t have followed them). Even if they somehow felt safe about that, there was no space for the complexity of their experiences. Listeners who didn’t exaggerate the goodness or badness of an experience still flattened complexity.
I suspect we disagree on defining academia rather than cult, which was still imprecise of me. Let’s take a humanities grad program
The only acceptable success criteria is being at the top of the pyramid within academia. People who quit for the outside world are disparaged and viewed as having wasted years of their life and others’.
You move have to move to a new location every few years and can’t afford to socialize with anyone but other academics.
You are probably paying a lot for the privilege of being there, often via debt.
Even people who start with stipends often find those run out before they finish, leaving them with the choice of abandoning 5 years of work or taking on debt for the last years.
If you’re not already from the right class, going to the program will cause you to change in ways that pull you from your family.
Depression/anxiety rates for grad students are sky high, and improve when they quit.
Your advisor can ruin your life with no consequences.
Members believe themselves to be special and answering a higher calling.
I agree that academia is less of a cult than the military, which can literally imprison you, but it’s on the spectrum.
Posts on cults I keep meaning to write:
Morpheus’s team in the Matrix could easily have been abusive and toxic, even though they were right about the whole world being fake. Correctness of world-model and interpersonal safety are unrelated axes.
Everyone I know who’s been in ~cults has found it to have significant upsides that the discourse made impossible to discuss in public. If you don’t personally know people who have been in ~cults you are probably underestimating the upside and complexity of the situations.
Some good things cannot be achieved without a high level of illiquid investment. High investment creates dynamics similar to high control. For certain purposes the difference between “the high cost of leaving is in the territory” and “someone has maliciously made leaving costly” is important, and for others its really not.
Vulnerability budgets- know yours and spend it wisely.
If a cult expert doesn’t recognize that
academia and[1] Catholic monasteries and the army are obviously cults, using the same techniques with the same harms, they are not trustworthy. I have yet to encounter an expert who recognizes this; usually they explicitly exempt old institutions.cult is an attractor state you have to actively fight to avoid
Partially because lots of people want to be in a cult and will actively push a leader into culthood against their will (or more likely, they are conflicted about it but would have held the line absent the pressure from followers). I give Eliezer a lot of credit for his refusal of this call.
Please distinguish between harms, harmless weirdnesses, and things that are harmless in and of themselves but create vulnerability to future harm.
Doing a lot of good initially is not totally protective against doing severe harm later on. If Jim Jones had died in Indiana he’d have been a regional civil rights hero, and if he’d died in California he’d be a complicated figure who did a lot of good, partially by providing a refuge for people escaping worse cults.
- ^
I crossed out academia 7h after posting because based on the comments this is only obvious to me, but I do still consider it obvious.
Did anyone end up taking you up on this?
It would have been so easy to meet Heaven’s Gate where it was coming from. Heaven’s Gate had a starting set of assumptions that you could crisply question. You could assert that there was no comet behind the spaceship, or that apparent-suicide wouldn’t allow you to board it.
My ideal version of the OP (that I am not owed) either explicitly says “Soryu/MAPLE is wrong about their factual claims that doom is imminent”, or agrees with that premise but gets into the details of how it believes MAPLE handled that fact badly. I don’t think MAPLE would come out any better in that version (I had a terrible experience there and public data says its been downhill from there), but it would feel like it left more room to recognize if a different organization had the same premise and handled implementation better.
As of this writing there are two top level comments on this post, both of which ask for more work. I agree with the requests and have a few of my own, which I’ve been planning to put in short form. But man, it is really annoying to put months of work into something and be met with more homework. I feel that way with science write-ups, and I can only imagine how much worse it feels when you’re talking about a multi-year traumatic experience. I think Kyle did a good job at the post he intended to write, it is valuable despite not fulfilling everyone’s wishlist, and I hope the requests for more don’t continue to dominate the conversation.
A few years ago I investigated how medieval Europeans built cathedrals. It was just incredible to me that anyone completed anything over multiple generations- how did anyone possibly stick with anything that long?
My conclusion after a fairly brief investigation:
they mostly didn’t. Many more cathedrals were attempted than finished.
a major contributor to the finished ones was a lack of competing options. Yes, people were poor, but given resources and a desire to spend them on something in the class of a cathedral, the local cathedral was really the only option.
I’m reminded of this every time I think about modern socialization. There were lots of reasons people participated in bowling leagues and rotary club, but one of them was that there wasn’t a lot else to do. You could lock in without worrying about the opportunity cost.
And that was true for everyone, and everyone knew it was true for everyone, which mean you could count on each others’ participation in the stag hunt, which made you and them even more willing to invest.
I didn’t play TTRPGs for ~10 years, but got back into over covid. I am supremely confident the game would have fallen apart if we’d had anything else to do. Some players were flakey even in lockdown, and it took a few months for the game to really gel. But by the end it really paid off and I was so glad I’d put in the time.
Trying to socialize in the bay area, especially stag-hunt socializing where people predictably invest and it takes time to pay off, is undercut by the sheer abundance of options. It’s hard to get people to cook a meal when there are so many snacks, even when every individual prefers meals to snacks.