Hm, this timing suggests the change could be a consequence of Karnofsky stepping away from the organization.
Which makes sense, now that I think about it. He’s by far the most politically strategic leader Open Philanthropy has had, so with him gone, it’s not shocking they might revert towards standard risk-averse optionality-maxxing foundation behavior.
“Open Phil higher-ups being friends with Anthropic higher-ups” is an understatement. An Open Philanthropy cofounder (Holden Karnofsky) is married to an Anthropic cofounder (Daniela Amodei). It’s a big deal!
I don’t recall Eliezer claiming that dath ilani characters never give in to threats. *Dath ilani characters* claim they never give in to threats. My interpretation is that the characters *say* “We don’t give in to threats”, and *believe* it, but it’s not *true*. Rather it’s something between a self-fulfilling prophecy, a noble lie-told-to-children, and an aspiration.
There are few threats in dath ilan, partly because the conceit of dath ilan is that it’s mostly composed of people who are cooperative-libertarianish by nature and don’t want to threaten each other very much, but partly because it’s a political structure where it’s much harder to get threats to actually *work*. One component of that political structure is how people are educated to defy threats by reflex, and to expect their own threats to fail, by learning am idealized system of game theory in which threats are always defied.
However, humans don’t actually follow ideal game theory when circumstances get sufficiently extreme, even dath ilani humans. Peranza can in fact be “shattered in Hell beyond all hope of repair” in the bad timeline, for all that she might rationally “decide not to break”. Similarly when the Head Keeper commits suicide to make a point: “So if anybody did deliberately destroy their own brain in attempt to increase their credibility—then obviously, the only sensible response would be to ignore that, so as not create hideous system incentives. Any sensible person would reason out that sensible response, expect it, and not try the true-suicide tactic.” But despite all that the government sets aside the obvious and sensible policy because, come on, the Head Keeper just blew up her own brain, stop fucking around and get serious. And the Head Keeper, who knows truths about psychology which the members of government do not, *accurately predicted they would respond that way*.
So dath ilani are educated to believe that giving in to threats is irrational, and to believe that people don’t give in to threats. This plus their legal system means that there are few threats, and the threats usually fail, so their belief is usually correct, and the average dath ilani never sees it falsified. Those who think carefully about the subject will realize that threats can sometimes work, in circumstances which are rare in dath ilan, but they’ll also realize that it’s antisocial to go around telling everyone about the limits of their threat-resistance and keep it quiet. The viewpoint characters start believing the dath ilani propaganda but update pretty quickly when removed from dath ilan. Keltham has little trouble understanding the Golarian equilibrium of force and threats once he gets oriented. Thellim presumably pays taxes off camera once she settles in to Earth.
>Are the near-term prospects of AGI making long-term prospects like suspension less attractive?
No. Everyone I know who was signed up for cryonics in 2014 is still signed up now. You’re hearing about it less because Yudkowsky is now doing other things with his time instead of promoting cryonics, and those discussions around here were a direct result of his efforts to constantly explain and remind people.
I agree with your argument here, especially your penultimate paragraph, but I’ll nitpick that framing your disagreements with Groves as him being “less of a value add” seems wrong. The value that Groves added was building the bomb, not setting diplomatic policy.
What is the mechanism, specifically, by which going slower will yield more “care”? What is the mechanism by which “care” will yield a better outcome? I see this model asserted pretty often, but no one ever spells out the details.
I’ve studied the history of technological development in some depth, and I haven’t seen anything to convince me that there’s a tradeoff between development speed on the one hand, and good outcomes on the other.
I’m coming to this late, but this seems weird. Do I understand correctly that many people were saying that Anthropic, the AI research company, had committed never to advance the state of the art of AI research, and they believed Anthropic would follow this commitment? That is just… really implausible.
This is the sort of commitment which very few individuals are psychologically capable of keeping, and which ~zero commercial organizations of more than three or four people are institutionally capable of keeping, assuming they actually do have the ability to advance the state of the art. I don’t know whether Anthropic leadership ever said they would do this, and if they said it then I don’t know whether they meant it earnestly. But even imagining they said it and meant it earnestly there is just no plausible world in which a company with hundreds of staff and billions of dollars of commercial investment would keep this commitment for very long. That is not the sort of thing you see from commercial research companies in hot fields.
If anyone here did believe that Anthropic would voluntarily refrain from advancing the state of the art in all cases, you might want to check if there are other things that people have told you about themselves, which you would really like to be true of them, but you have no evidence for other than their assertions, and would be very unusual if they were true.
Ben is working on a response, and given that I think it’s clearly the right call to wait a week or two until we have another round of counter-evidence before jumping to conclusions. If in a week or two people still think the section of “Avoidable, Unambiguous falsehoods” does indeed contain such things, then I think an analysis like this makes sense
This was three months ago. I have not seen the anticipated response. Setting aside the internal validity of your argument above, the promised counterevidence did not arrive in anything like a reasonable time.
TracingWoodgrains clearly made the right call in publishing, rather than waiting for you.
Yes, obviously, but they use different strategies. Male sociopaths rarely paint themselves as helpless victims because it is not an effective tactic for men. One does notice that, while the LW community is mostly male, ~every successful callout post against a LW community organization has been built on claims of harm to vulnerable female victims.
When you say “it’s clearly the right call to wait a week or two until we have another round of counter-evidence before jumping to conclusions”, is this a deliberate or accidental echo of the similar request from Nonlinear which you denied?
Like, on the deliberate way of reading this, the subtext is “While Lightcone did not wait a week or two for counter-evidence and still defends this decision, you should have waited in your case because that’s the standard you describe in your article.” Which would be a hell of a thing to say without explicitly acknowledging that you’re asking for different standards. (And would also misunderstand TracingWoodgrains’s actual standard, which is about the algorithm used and not how much clock time is elapsed, as described in their reply to your parent comment.) Or on the accidental way of reading this, the subtext is “I was oblivious to how being publicly accused of wrongdoing feels from the inside, and I request grace now that the shoe is on the other foot.” Either of these seems kind of incredible but I can’t easily think of another plausible way of reading this. I suppose your paragraph on wanting to take the time to make a comprehensive response (which I agree with) updates my guess towards “oblivious”.
On Pace’s original post I wrote:
“think about how bad you expect the information would be if I selected for the worst, credible info I could share”
Alright. Knowing nothing about Nonlinear or about Ben, but based on the rationalist milieu, then for an org that’s weird but basically fine I’d expect to see stuff like ex-employees alleging a nebulously “abusive” environment based on their own legitimately bad experiences and painting a gestalt picture that suggests unpleasant practices but without any smoking-gun allegations of really egregious concrete behavior (as distinct from very bad effects on the accusers); allegations of nepotism based on social connections between the org’s leadership and their funders or staff; accusations of shoddy or motivated research which require hours to evaluate; sources staying anonymous for fear of “retaliation” but without being able to point to any legible instances of retaliation or concrete threats to justify this; and/or thirdhand reports of lying or misdirection around complicated social situations.
[reads post]
This sure has a lot more allegations of very specific and egregious behavior than that, yeah.
Having looked at the evidence and documentation which Nonlinear provides, it seems like the smoking-gun allegations of really egregious concrete behavior are probably just false. I have edited my earlier comment accordingly.
This is a bit of a tangent, but is there a biological meaning to the term “longevity drug”? For a layman like me, my first guess is that it’d mean something like “A drug that mitigates the effects of aging and makes you live longer even if you don’t actively have a disease to treat.” But then I’d imagine that e.g. statins would be a “longevity drug” for middle-aged men with a strong family history of heart disease, in that it makes the relevant population less susceptible to an aging-related disease and thereby increases longevity, yet the posts talk about the prospect of creating the “first longevity drug” so clearly it’s being used in a way that doesn’t include statins. Is there a specific definition I’m ignorant of, or is it more of a loose marketing term for a particular subculture of researchers and funders, or what?
We can certainly debate whether liability ought to work this way. Personally I disagree, for reasons others have laid out here, but it’s fun to think through.
Still, it’s worth saying explicitly that as regards the motivating problem of AI governance, this is not currently how liability works. Any liability-based strategy for AI regulation must either work within the existing liability framework, or (much less practically) overhaul the liability framework as its first step.
Cars are net positive, and also cause lots of harm. Car companies are sometimes held liable for the harm caused by cars, e.g. if they fail to conform to legal safety standards or if they sell cars with defects. More frequently the liability falls on e.g. a negligent driver or is just ascribed to accident. The solution is not just “car companies should pay out for every harm that involves a car”, partly because the car companies also don’t capture all or even most of the benefits of cars, but mostly because that’s an absurd overreach which ignores people’s agency in using the products they purchase. Making cars (or ladders or knives or printing presses or...) “robust to misuse”, as you put it, is not the manufacturer’s job.
Liability for current AI systems could be a good idea, but it’d be much less sweeping than what you’re talking about here, and would depend a lot on setting safety standards which properly distinguish cases analogous to “Alice died when the car battery caught fire because of poor quality controls” from cases analogous to “Bob died when he got drunk and slammed into a tree at 70mph”.
It’s fun to come through and look for interesting threads to pull on. I skim past most stuff but there’s plenty of good and relevant writing to keep me coming back. Yeah sure it doesn’t do a super great job of living up to the grandiose ideals expressed in the Sequences but I don’t really mind, I don’t feel invested in ~the community~ that way so I’ll gladly take this site for what it is. This is a good discussion forum and I’m glad it’s here.
Toner’s employer, the Center for Security and Emerging Technology (CSET), was founded by Jason Matheny. Matheny was previously the Director of the Intelligence Advanced Research Projects Activity (IARPA), and is currently CEO of the RAND Corporation. CSET is currently led by Dewey Murdick, who previously worked at the Department of Homeland Security and at IARPA. Much of CSET’s initial staff was former (or “former”) U.S. intelligence analysts, although IIRC they were from military intelligence rather than the CIA specifically. Today many of CSET’s researchers list prior experience with U.S. civilian intelligence, military intelligence, or defense intelligence contractors. Given the overlap in staff and mission, U.S. intelligence clearly and explicitly has a lot of influence at CSET, and it’s reasonable to suspect a stronger connection than that.
I don’t see it for McCauley though.
Suppose you’re an engineer at SpaceX. You’ve always loved rockets, and Elon Musk seems like the guy who’s getting them built. You go to work on Saturdays, you sometimes spend ten hours at the office, you watch the rockets take off and you watch the rockets land intact and that makes everything worth it.
Now imagine that Musk gets in trouble with the government. Let’s say the Securities and Exchange Commission charges him with fraud again, and this time they’re *really* going after him, not just letting him go with a slap on the wrist like the first time. SpaceX’s board of directors negotiates with SEC prosecutors. When they emerge they fire Musk from SpaceX, and remove Elon and Kimbal Musk from the board. They appoint Gwynne Shotwell as the new CEO.
You’re pretty worried! You like Shotwell, sure, but Musk’s charisma and his intangible magic have been very important to the company’s success so far. You’re not sure what will happen to the company without him. Will you still be making revolutionary new rockets in five years, or will the company regress to the mean like Boeing? You talk to some colleagues, and they’re afraid and angry. No one knows what’s happening. Alice says that the company would be nothing without Musk and rails at the board for betraying him. Bob says the government has been going after Musk on trumped-up charges for a while, and now they finally got him. Rumor has it that Musk is planning to start a new rocket company.
Then Shotwell resigns in protest. She signs an open letter calling for Musk’s reinstatement and the resignation of the board. Board member Luke Nosek signs it too, and says his earlier vote to fire Musk was a huge mistake.
You get a Slack message from Alice saying that she’s signed the letter because she has faith in Musk and wants to work at his company, whichever company that is, in order to make humanity a multiplanetary species. She asks if you want to sign.
How do you feel?
I really don’t think you can justify putting this much trust in the NYT’s narrative of events and motivations here. Like, yes, Toner did publish the paper, and probably Altman did send her an email about it. Then the NYT article tacitly implies but *doesn’t explicitly say* this was the spark that set everything off, which is the sort of haha-it’s-not-technically-lying that I expect from the NYT. This post depends on that implication being true.
Yeah it’s pretty bad. There are some professors who are better than just reading the textbook, but unfortunately they’re the exceptions. My undergrad experience got a lot more productive once I started picking my courses based on the *teacher* more than on the *subject*.
What does your company do, specifically? I found the brief description at HealthcareAgents.com vague and unclear. Can you walk me through an example case of what you do for a patient, or something?