should take the reputational costs of your decisions into account
Specifically, you should take the costs of people applying generally-good rules of evaluation into account. The arguments for taking people’s bad rules of evaluation (e.g. if people were freaking out because Caroline was a woman) are much narrower and the right default is fuck ’em.
You’re pointing to “I am evaluating Austin as doing something bad given the limited information that I have”, whereas Elizabeth seems to be getting at something more like “it is Austin’s job (as a member of this ecosystem) to give us more information when he does something like this; and he’s failing at his job”.
I’m probably closer to the latter than Ben is but it’s not how I would phrase it and it’s a somewhat separate principle.
I wouldn’t use the naked phrase “it is Austin’s job to”. But I would say “if Austin doesn’t want to be judged as a central example of hiring a fraud felon, the burden is on him to make it sufficiently cheap to detect that Caroline is different (which may be impossible, and that’s not a failure of the observers).” The difference is that Austin is still allowed to decide hiring Caroline is worth the judgement it incurs. I’m just not sympathetic to the consequences, even if a burning bush really did vouch for Caroline.
even if you’re not doing shady shit you’re being irresponsible by not giving outside observers enough information to be confident that you’re not doing shady shit”
The legal term here is “demonstrable compliance”, and I think it’s really important. It’s not enough to know how to drive, you have to prove it to the state, because a system where everyone judges their own driving leads to too many deaths. It’s not enough to buy a ticket to the conference, you need to wear your badge so they can see you did so.
TBC my core objection is “Caroline is in the class of people for which we accurately expect a higher rate of fraud”, and I’m trying to explain why I think that that is sufficient for people to update on Austin’s fraud tolerance and to consider those updates in their decisions, even if a burning bush tells Austin privately that Caroline is an exception.
I can’t see a clear line between what is happening here and what I consider the primary causes of the FTX fallout. When I wanted to talk openly about Sam being shady, people said “it would look really bad if you publicly accused a major leader of EA of being shady without having ironclad evidence of backing that up, and this seems like a pretty clear rule”. Please, can we stop doing this.
I agree that that there’s not a clear line, in the sense that I don’t get how you’re connecting them at all. Obviously I wished you’d shared those concerns.
Are there status/class implications as well?
“Bad optics” usually means an action is violating some rule.
Sometimes the rule is good, and the problem is that you are an idiot violating a good rule.
Sometimes that rule is bad and it’s good to break it. There’s a lot of nobility in doing what you believe is right and accepting the consequences. And there’s a lot of good in a culture that does what they think is right without concern for public opinion, even if this comes with a bunch of offputting stuff that makes their job harder.
Sometimes the rule is good, but if people paid enough attention they would realize it does not apply in your particular case. But people don’t owe you infinite attention. Depending on the specifics it can be fair for people to round you to scenario 1 and proceed accordingly. “But I’m right” is not a defense in this case even to people who agree with you that your case is an exception, in part because disregarding the rule weakens it, and that’s bad because we’ve agreed the rule is good. So sometimes you should obey a rule even if you know for sure it doesn’t apply, to stay legibly good to others.
For example, I think “don’t put financial crime felons in charge of large amounts of money” is a good rule. It’s not a perfect rule- some felons learned their moral lesson, or just aren’t going to have the same opportunity to steal. But overall we as a society are better off with that rule, and I’m comfortable with the collateral damage of people who commit financial crimes having worse jobs later.
A lot of the criticism of Austin Chen hiring Caroline Ellison has focused on the fact that people will judge EA for it, and Austin’s defense has been that she’s really good at the job and has learned her lesson, with an implicit claim that people the people judging EA for this hire are wrong. But that’s insufficient- this is scenario 3, and you need to prove not only that she is an exception, but that detecting this is sufficiently cheap. He hasn’t tried to do that, and I don’t think it’s possible. So while I’m normally disgusted by EA calls to optimize for optics, in this case I think they’re right.
A few years ago I investigated how medieval Europeans built cathedrals. It was just incredible to me that anyone completed anything over multiple generations- how did anyone possibly stick with anything that long?
My conclusion after a fairly brief investigation:
they mostly didn’t. Many more cathedrals were attempted than finished.
a major contributor to the finished ones was a lack of competing options. Yes, people were poor, but given resources and a desire to spend them on something in the class of a cathedral, the local cathedral was really the only option.
I’m reminded of this every time I think about modern socialization. There were lots of reasons people participated in bowling leagues and rotary club, but one of them was that there wasn’t a lot else to do. You could lock in without worrying about the opportunity cost.
And that was true for everyone, and everyone knew it was true for everyone, which mean you could count on each others’ participation in the stag hunt, which made you and them even more willing to invest.
I didn’t play TTRPGs for ~10 years, but got back into over covid. I am supremely confident the game would have fallen apart if we’d had anything else to do. Some players were flakey even in lockdown, and it took a few months for the game to really gel. But by the end it really paid off and I was so glad I’d put in the time.
Trying to socialize in the bay area, especially stag-hunt socializing where people predictably invest and it takes time to pay off, is undercut by the sheer abundance of options. It’s hard to get people to cook a meal when there are so many snacks, even when every individual prefers meals to snacks.
The summary judgement doesn’t cite Dean in particular but does cite the amicus brief and number of signers twice (search for “176”). Seems like Jeff Dean’s name may not matter much but raising the # of signers by 30% might have?
It seems like you’re assuming that work labeled as safety is safety, or at least more safety than capabilities.
In the ancestral environment, infection was a top killer. Today it’s a rounding error. Our immune systems are therefore set by evolution to be far too sensitive for the modern environment.
Definitely for certain kinds of diseases (macroparasites, anything where humans aren’t the primary resevoir), maybe for others (septicemia? things spread by water or mosquito?) but e.g. respiratory illnesses are much more abundant now.
There exist watermarking schemes such that it is completely impossible for users to ever distinguish between watermarked and non-watermarked text. (A tempting but invalid argument is that no such scheme can exist because watermarking consumes entropy.)
I agree that this is the crux, but have yet to see anyone making this claim provide evidence or an explanation as to how it is possible.
People like brevity but will punish authors for leaving things out so authors defend themselves with length.
Because some people need to set “Reign of Terror” in order to feel comfortable doing any moderation at all.
Additionally, Wei Dai had not set Reign of Terror when the comments were made.
See the ensuing conversation with Habryka. I assumed this because people claimed it, and MIRI’s defenders responded with “this does not meet the technical definition of paying off aggressors”, and not the straightforward “MIRI never said they wouldn’t pay off aggressors”. Sounds like I was wrong, but to this day that response only makes sense to me if the writer is conceding that MIRI committed to never paying blackmail.
I think it would help to fully explain the chain of thought risk in this post, even if it’s redundant with the link. Most people don’t click links, and it’s an important argument.
The big take home from @TurnTrout’s post on his lobbying efforts within google is that they didn’t work, Google signed an “all lawful use” contract with the pentagon. But he did (reportedly) get Jeff Dean to sign an amicus brief in Anthropic’s lawsuit agains the pentagon. Does anyone know what the value of that is? If it gets Anthropic marginally more damages, or more protection from a designation as a supply chain risk, that seems value-creating.
Pretty much anyone in academia will tell prospective students “don’t do a phd” or at least “you should only do a phd if narrow criteria are met
My undergrad professors heavily encouraged me to reapply to grad school (in biology) because I was so special and good.
In hindsight, given the poor track record of large alignment-branded research organizations, it was a significant mistake for me to accept Resolution’s offer without having thought much more carefully about ways that joining might be bad (such as the points above). There’s a common failure mode whereby rationalists and EAs are averse to thinking about how their cognition is skewed by incentives and power dynamics; though I often talk about this issue abstractly, I still failed to account for it in this case.
I’d be very interested in a 5-whys on this.
This seems so much more exploitable and closer to respectability politics than anything I said