I really don’t think you can justify putting this much trust in the NYT’s narrative of events and motivations here. Like, yes, Toner did publish the paper, and probably Altman did send her an email about it. Then the NYT article tacitly implies but *doesn’t explicitly say* this was the spark that set everything off, which is the sort of haha-it’s-not-technically-lying that I expect from the NYT. This post depends on that implication being true.
David Hornbein
Karma: 1,175
The Unilateralist’s “Curse” Is Mostly Good
When you say “it’s clearly the right call to wait a week or two until we have another round of counter-evidence before jumping to conclusions”, is this a deliberate or accidental echo of the similar request from Nonlinear which you denied?
Like, on the deliberate way of reading this, the subtext is “While Lightcone did not wait a week or two for counter-evidence and still defends this decision, you should have waited in your case because that’s the standard you describe in your article.” Which would be a hell of a thing to say without explicitly acknowledging that you’re asking for different standards. (And would also misunderstand TracingWoodgrains’s actual standard, which is about the algorithm used and not how much clock time is elapsed, as described in their reply to your parent comment.) Or on the accidental way of reading this, the subtext is “I was oblivious to how being publicly accused of wrongdoing feels from the inside, and I request grace now that the shoe is on the other foot.” Either of these seems kind of incredible but I can’t easily think of another plausible way of reading this. I suppose your paragraph on wanting to take the time to make a comprehensive response (which I agree with) updates my guess towards “oblivious”.
Yes, obviously, but they use different strategies. Male sociopaths rarely paint themselves as helpless victims because it is not an effective tactic for men. One does notice that, while the LW community is mostly male, ~every successful callout post against a LW community organization has been built on claims of harm to vulnerable female victims.
I disagree strongly. To me it seems that AI safety has long punched below its weight because its proponents are unwilling to be confrontational, and are too reluctant to put moderate social pressure on people doing the activities which AI safety proponents hold to be very extremely bad. It is not a coincidence that among AI safety proponents, Eliezer is both unusually confrontational and unusually successful.
This isn’t specific to AI safety. A lot of people in this community generally believe that arguments which make people feel bad are counterproductive because people will be “turned off”.
This is false. There are tons of examples of disparaging arguments against bad (or “bad”) behavior that succeed wildly. Such arguments very frequently succeed in instilling individual values like e.g. conscientiousness or honesty. Prominent political movements which use this rhetoric abound. When this website was young, Eliezer and many others participated in an aggressive campaign of discourse against religious ideas, and this campaign accomplished many of its goals. I could name many many more large and small examples. I bet you can too.
Obviously this isn’t to say that confrontational and insulting argument is always the best style. Sometimes it’s truth-tracking and sometimes it isn’t. Sometimes it’s persuasive and sometimes it isn’t. Which cases are which is a difficult topic that I won’t get into here (except to briefly mention that it matters a lot whether the reasons given are actually good). Nor is this to say that the “turning people off” effect is completely absent; what I object to is the casual assumption that it outweighs any other effects. (Personally I’m turned off by the soft-gloved style of the parent comment, but I would not claim this necessarily means it’s inappropriate or ineffective—it’s not directed at me!) The point is that this very frequent claim does not match the evidence. Indeed, strong counterevidence is so easy to find that I suspect this is often not people’s real objection.
While “Vassar’s group” is informal, it’s more than just a cluster of friends; it’s a social scene with lots of shared concepts, terminology, and outlook (although of course not every member holds every view and members sometimes disagree about the concepts, etc etc). In this way, the structure is similar to social scenes like “the AI safety community” or “wokeness” or “the startup scene” that coordinate in part on the basis of shared ideology even in the absence of institutional coordination, albeit much smaller. There is no formal institution governing the scene, and as far as I’ve ever heard Vassar himself has no particular authority within it beyond individual persuasion and his reputation.
Median Group is the closest thing to a “Vassarite” institution, in that its listed members are 2⁄3 people who I’ve heard/read describing the strong influence Vassar has had on their thinking and 1⁄3 people I don’t know, but AFAIK Median Group is just a project put together by a bunch of friends with similar outlook and doesn’t claim to speak for the whole scene or anything.
I don’t understand the logic of this. Does seem like game-theoretically the net-payout is really what matters. What would be the argument for something else mattering?
BEORNWULF: A messenger from the besiegers!
WIGMUND: Send him away. We have nothing to discuss with the norsemen while we are at war.
AELFRED: We might as well hear them out. This siege is deadly dull. Norseman, deliver your message, and then leave so that we may discuss our reply.
MESSENGER: Sigurd bids me say that if you give us two thirds of the gold in your treasury, our army will depart. He reminds you that if this siege goes on, you will lose the harvest, and this will cost you more dearly than the gold he demands.
The messenger exits.
AELFRED: Ah. Well, I can’t blame him for trying. But no, certainly not.
BEORNWULF: Hold on, I know what you’re thinking, but this actually makes sense. When Sigurd’s army first showed up, I was the first to argue against paying him off. After all, if we’d paid right at the start, then he would’ve made a profit on the attack, and it would only encourage more. But the siege has been long and hard for us both. If we accept this deal *now*, he’ll take a net loss. We’ve spent most of the treasury resisting the siege—
WIGMUND: As we should! Millions for defense, but not one cent for tribute!
BEORNWULF: Certainly. But the gold we have left won’t even cover what they’ve already spent on their attack. Their net payout will still be negative, so game-theoretically, it doesn’t make sense to think of it as “tribute”. As long as we’re extremely sure they’re in the red, we should minimize our own costs, and missing the harvest would be a *huge* cost. People will starve. The deal is a good one.
WIGMUND: Never! if once you have paid him the danegeld, you never get rid of the Dane!
BEORNWULF: Not quite. The mechanism matters. The Dane has an incentive to return *only if the danegeld exceeds his costs*.
WIGMUND: Look, you can mess with the categories however you like, and find some clever math that justifies doing whatever you’ve already decided you want to do. None of that constrains your behavior and so none of that matters. What matters is, take away all the fancy definitions and you’re still just paying danegeld.
BEORNWULF: How can I put this in language you’ll understand—it doesn’t matter whether the definitions support what *I* want to do, it matters whether the definitions reflect the *norsemen’s* decision algorithm. *They* care about the net payout, not the gross payout.
AELFRED: Hold on. Are you modeling the norsemen as profit-maximizers?
BEORNWULF: More or less? I mean, no one is perfectly rational, but yeah, everyone *approximates* a rational profit-maximizer.
WIGMUND: They are savage, irrational heathens! They never even study game theory!
BEORNWULF: Come on. I’ll grant that they don’t use the same jargon we do, but they attack because they expect to make a profit off it. If they don’t expect to profit, they’ll stop. Surely they do *that* much even without explicit game theoretic proofs.
AELFRED: That affects their decision, yes, but it’s far from the whole story. The norsemen care about more than just gold and monetary profit. They care about pride. Dominance. Social rank and standing. Their average warrior is a young man in his teens or early twenties. When he decides whether to join the chief’s attack, he’s not sitting down with spreadsheets and a green visor to compute the expected value, he’s remembering that time cousin Guthrum showed off the silver chalice he looted from Lindisfarne. Remember, Sigurd brought the army here in the first place to avenge his brother’s death—
BEORNWULF: That’s a transparent pretext! He can’t possibly blame us for that, we killed Agnarr in self-defense during the raid on the abbey.
WIGMUND: You can tell that to Sigurd. If it had been my brother, I’d avenge him too.
AELFRED: Among their people, when a man is murdered, it’s not a *tragedy* to his family, it’s an *insult*. It can only be wiped away with either a weregeld payment from the murderer or a blood feud. Yes, Sigurd cares about gold, but he also cares tremendously about *personally knowing he defeated us*, in order to remove the shame we dealt him by killing Agnarr. Modeling his decisions as profit-maximizing will miss a bunch of his actual decision criteria and constraints, and therefore fail to predict the norsemen’s future actions.
WIGMUND: You’re overcomplicating this. If we pay, the norsemen will learn that we pay, and more will come. If we do not pay, they will learn that we do not pay, and fewer will come.
BEORNWULF: They don’t care if we *pay*, they care if it’s *profitable*. This is basic accounting.
AELFRED: They *do* care if we pay. Most of them won’t know or care what the net-payout is. If we pay tribute, this will raise Sigurd’s prestige in their eyes no matter how much he spent on the expedition, and he needs his warriors’ support more than he needs our gold. Taking a net loss won’t change his view on whether he’s avenged the insult to his family, and we do *not* want the Norsemen to think they can get away with coming here to avenge “insults” like killing their raiders in self-defense. On the other hand, if Sigurd goes home doubly shamed by failing to make us submit, they’ll think twice about trying that next time.
BEORNWULF: I don’t care about insults. I don’t care what Sigurd’s warriors think of him. I don’t care who can spin a story of glorious victory or who ends up feeling like they took a shameful defeat. I care about how many of our people will die on norse spears, and how many of our people will die of famine if we don’t get the harvest in. All that other stuff is trivial bullshit in comparison.
AELFRED: That all makes sense. You still ought to track those things instrumentally. The norsemen care about all that, and it affects their behavior. If you want a model of how to deter them, you have to model the trivial bullshit that they care about. If you abstract away what they *actually do* care about with a model of what you think they *ought* to care about, then your model *won’t work*, and you might find yourself surprised when they attack again because they correctly predict that you’ll cave on “trivial bullshit”. Henry IV could swallow his pride and say “Paris is well worth a mass”, but that was because he was *correctly modeling* the Parisians’ pride.
WIGMUND: Wait. That is *wildly* anachronistic. Henry converted to Catholicism in 1593. This dialogue is taking place in, what, probably the 9th century?
AELFRED: Hey, I didn’t make a fuss when you quoted Kipling.
Offering a specific amount of pay, in cash and in kind, and then not doing the accounting to determine whether or not that amount was actually paid out. If I’m charitable to the point of gullibility, then this is unethical and culpable negligence. Probably it’s just fraud. (Assuming this allegation is true, of course, and
AFAIK it is not yet disputed.)Screenshots of threats to retaliate for speaking up.
EDIT: Nonlinear has now replied and disputed many of the allegations. I am persuaded that the allegation of fraud/negligence around payment is simply false. As for the screenshots of threats to retaliate, my opinion is that retaliation or threats to retaliate are perfectly justified in the face of the behavior which Nonlinear alleges. Nonlinear also provides longer chatlogs around one of the screenshotted texts which they argue recontextualizes it.
I’ve had very good results from offering unsolicited no-strings $X,000 gifts to friends who were financially struggling while doing important work. Some people have accepted, and it took serious pressure off them while they laid the foundations for what’s become impressive careers. Some turned me down, although I like to imagine that knowing they had backup available made them feel like they had more slack. It’s a great way to turn local social knowledge into impact. The opportunities to do this aren’t super frequent, but when it arises the per-dollar impact is absolutely insane.
Some advice for anyone who’s thinking of doing this:
—Don’t mention it to anyone else. This is sensitive personal stuff. Your friend doesn’t want some thirdhand acquaintance knowing about their financial hardship. Talking about it in general terms without identifying information is fine.
—Make gifts, not loans. Loans often put strain on relationships and generally make things weird. People sometimes make “loans” they can’t afford to lose without realizing what they’re doing, but you’re not gonna give away money without understanding the consequences. Hold fast to this; if the recipient comes back three years later when they’re doing better and offers to pay it back (this has happened to me), tell them to pay it forward instead.
—The results will only be as good as your judgements of character and ability. Everyone makes mistakes, but if you make *big* mistakes on either of those, then this probably isn’t your comparative advantage.
My guess: Sutskever was surprised by the threatened mass exodus. Whatever he originally planned to achieve, he no longer thinks he can succeed. He now thinks that falling on his sword will salvage more of what he cares about than letting the exodus happen.
I’m coming to this late, but this seems weird. Do I understand correctly that many people were saying that Anthropic, the AI research company, had committed never to advance the state of the art of AI research, and they believed Anthropic would follow this commitment? That is just… really implausible.
This is the sort of commitment which very few individuals are psychologically capable of keeping, and which ~zero commercial organizations of more than three or four people are institutionally capable of keeping, assuming they actually do have the ability to advance the state of the art. I don’t know whether Anthropic leadership ever said they would do this, and if they said it then I don’t know whether they meant it earnestly. But even imagining they said it and meant it earnestly there is just no plausible world in which a company with hundreds of staff and billions of dollars of commercial investment would keep this commitment for very long. That is not the sort of thing you see from commercial research companies in hot fields.
If anyone here did believe that Anthropic would voluntarily refrain from advancing the state of the art in all cases, you might want to check if there are other things that people have told you about themselves, which you would really like to be true of them, but you have no evidence for other than their assertions, and would be very unusual if they were true.
Seems also worth mentioning that Gretta Duleba and Eliezer Yudkowsky are in a romantic relationship, according to Yudkowsky’s Facebook profile.
I would expand “acts to make the argument low status” to “acts to make the argument low status without addressing the argument”. Lots of good rationalist material, including the original Sequences, includes a fair amount of “acts to make arguments low status”. This is fine—good, even—because it treats the arguments it targets in good faith and has a message that rhymes with “this argument is embarrassing because it is clearly wrong, as I have shown in section 2 above” rather than “this argument is embarrassing because gross stupid creeps believe it”.
Many arguments are actually very bad. It’s reasonable and fair to have a lower opinion of people who hold them, and to convey that opinion to others along with the justification. As you say, “you shouldn’t engage by not addressing the arguments and instead trying to execute some social maneuver to discredit it”. Discrediting arguments by social maneuvers that rely on actual engagement with the argument’s contents is compatible with this.
If you make programmer money and a bunch of your friends are working on weird projects that take two hours to explain and justify—and I know that describes a lot of people here—then you’re in an excellent position to do this. Essentially it’s legibility arbitrage.
For this reason, I wouldn’t want this post included in the 2019 highlights. I just looked at this for the review, and the part which some people report finding useful is in the brief description of the concept at the very beginning. The bulk of the post is a freeform, rambling exploration of the concept and its implications which I mostly couldn’t bring myself to focus on; this exploratory style seems totally appropriate for a personal blog post, but it’s not the sort of thing I’d want to read if I were looking back at a curated list of the best stuff from 2019.
On Pace’s original post I wrote:
“think about how bad you expect the information would be if I selected for the worst, credible info I could share”
Alright. Knowing nothing about Nonlinear or about Ben, but based on the rationalist milieu, then for an org that’s weird but basically fine I’d expect to see stuff like ex-employees alleging a nebulously “abusive” environment based on their own legitimately bad experiences and painting a gestalt picture that suggests unpleasant practices but without any smoking-gun allegations of really egregious concrete behavior (as distinct from very bad effects on the accusers); allegations of nepotism based on social connections between the org’s leadership and their funders or staff; accusations of shoddy or motivated research which require hours to evaluate; sources staying anonymous for fear of “retaliation” but without being able to point to any legible instances of retaliation or concrete threats to justify this; and/or thirdhand reports of lying or misdirection around complicated social situations.
[reads post]
This sure has a lot more allegations of very specific and egregious behavior than that, yeah.
Having looked at the evidence and documentation which Nonlinear provides, it seems like the smoking-gun allegations of really egregious concrete behavior are probably just false. I have edited my earlier comment accordingly.
Registering that I much prefer the format of the older repositories you link to, where additions are left as comments that can be voted on, over the format here, where everything is in a giant list sorted by topic rather than ranking. For any crowdsourced repository, most suggestions will be mediocre or half-baked, but with voting and sorting it’s easy to read only the ones that rise to the top. I’d also be curious to check out the highest-voted suggestions on this topic, but not curious enough to wade through an unranked list of (I assume) mostly mediocre and half-baked ideas to find them.
I’ve actually done this and it worked incredibly well, so I’m not persuaded by your vague doubts that it’s possible. If you insist on using “opinion of established organizations” as your yard stick then I’ll add that a strong majority of the people I supported would later go on to get big grants and contracts from well-respected organizations, always after years of polishing and legibilizing the projects which I’d supported in their infancy.
Certainly it wouldn’t work for the median person on Earth. But “LW reader who’s friends with a bunch of self-starting autodidacts and has enough gumption to actually write a check” is not the median person on Earth, and people selected that hard will often know some pretty impressive people.
Alright. Knowing nothing about Nonlinear or about Ben, but based on the rationalist milieu, then for an org that’s weird but basically fine I’d expect to see stuff like ex-employees alleging a nebulously “abusive” environment based on their own legitimately bad experiences and painting a gestalt picture that suggests unpleasant practices but without any smoking-gun allegations of really egregious concrete behavior (as distinct from very bad effects on the accusers); allegations of nepotism based on social connections between the org’s leadership and their funders or staff; accusations of shoddy or motivated research which require hours to evaluate; sources staying anonymous for fear of “retaliation” but without being able to point to any legible instances of retaliation or concrete threats to justify this; and/or thirdhand reports of lying or misdirection around complicated social situations.
[reads post]
This sure has a lot more allegations of very specific and egregious behavior than that, yeah.
EDIT: Based on Nonlinear’s reply and the thorough records they provide, it seems that the smoking-gun allegations of really egregious concrete behavior are probably just false. This leaves room for unresolvable disagreement on the more nebulous accusations, but as I said initially, that’s the pattern I’d expect to see if Nonlinear were weird but basically fine.