Orpheus16

Karma: 6,905

Orpheus16 4 Jun 2026 17:48 UTC
33 points
4
on: Akash’s Shortform
Anthropic releases new post about RSI and slowdowns—also commits to doing verification research and verification-focused policymaker engagement. (In my view, this is probably the best post/announcement about AI risks that has been produced by any frontier AI company in recent memory).

https://www.anthropic.com/institute/recursive-self-improvement
We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.
A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions. It would also require that each can verify that the others have actually stopped. Due to the unique characteristics of AI systems, the detectability (a lower standard than verifiability) element of this arms control problem is much more challenging than with other technologies. Training runs are far easier to conceal than missile silos, their inputs are general-purpose, and the incentive to defect quietly is enormous, because whoever continues while others pause could inherit the lead. A credible pause also has to specify what triggers it, what lifts it, and who adjudicates.
None of this is necessarily impossible in principle—the world has built verification regimes for other complex technologies (e.g., the Intermediate-Range Nuclear Forces Treaty)—but those regimes took decades to build both the infrastructure and the trust. We don’t have that long. A unilateral pause by one lab, by contrast, is achievable immediately, but accomplishes much less: it would change who the front-runner is, but it would not create the wider deliberative process that is currently missing.
In the coming months, we will organize conversations where policymakers, researchers, civil society, and other AI companies can help answer some of the questions this piece raises, especially around full recursive self-improvement and how to create better options for coordination and deliberation. We’ll publish what comes out of it. The window to investigate the questions together is here, and people outside AI companies should be involved in this deliberation.

Orpheus16 4 Nov 2025 2:40 UTC
41 points
28
on: Leaving Open Philanthropy, going to Anthropic
I think it would be valuable to ask Anthropic’s policy team (and/or leadership) if they agree with these statements (or adjacent statements), and if they have any plans to prioritize these kinds of statements in their communications with policymakers & the public.
It seems to me like a lot of Anthropic employees agree with these statements (or adjacent statements), yet this does not appear to be guiding Anthropic’s official lobbying or policy activities.
I think that the technology being built by companies like Anthropic has a significant (read: double-digit) probability of destroying the entire future of the human species.
What’s more, I think no private company should be in a position to impose this kind of risk on every living human, and I support efforts to make sure that no company ever is.
Further: I do not think that Anthropic or any other actor has an adequate plan for building superintelligence in a manner that brings the risk of catastrophic, civilization-ending misalignment to a level that a prudent and coordinated civilization would accept.
More specifically: I do not believe that the object-level benefits of advanced AI^[18] – serious though they may be – currently justify the level of existential risk at stake in any actor, Anthropic included, developing superintelligence given our current understanding of how to do so safely.^[19]
But there is, indeed, a clear solution to this problem in principle: namely, to use various methods of capability restraint (coordination, enforcement, etc) to ensure that no one develops superintelligence until we have a radically better understanding of how to do so safely.
I have no idea how Anthropic’s policy team makes decisions, but insofar as they value the input of employees on other teams, it seems plausible to me that Anthropic employees with these beliefs (or adjacent beliefs) could play a meaningful role by speaking out about these beliefs, requesting more information about Anthropic’s policy engagements, and having more discussions with Anthropic policy/leadership teams about if/how Anthropic could prioritize these topics more in its policy work & public comms.

Orpheus16 18 Oct 2025 16:17 UTC
6 points
0
in reply to: Garrett Baker’s comment on: Mikhail Samin’s Shortform
I’m not sure if we disagree— I think there are better ways to assess this than the way the “is this an xrisk person or not” tribal card often gets applied.
- Example: “Among all the topics in AI policy and concerns around AI, what are your biggest priorities?” is a good question IMP.
- Counterexample: “Do you think existential risk from advanced AI is important?” is a bad question IMO (especially in isolation).
It is very easy for people to say they care about “AI safety” without giving much indication of where it stands on their priority list, what sorts of ideas/plans they want to aim for, what threat models they are concerned about, if they are the kind of person who can have a 20+ min conversation about interesting readings or topics in the field, etc.
I suspect that people would get “burnt” less if they asked these kinds of questions instead of defaulting to some sort of “does this person care about safety” frame or “is this person Part of My Tribe” thing.
(On that latter point, it is rather often that I hear people say things like “Alice is amazing!” and then when I ask them about Alice’s beliefs or work they say something like “Oh I don’t know much about Alice’s work— I just know other people say Alice is amazing!”. I think it would be better for people to say “I think Alice is well-liked but I personally do not know much about her work or what kinds of things she believes/prioritizes.”)
What links here?
- Garrett Baker's comment on Mikhail Samin’s Shortform by Mikhail Samin (21 Oct 2025 0:59 UTC; 2 points)

Orpheus16 17 Oct 2025 22:01 UTC
19 points
0
in reply to: Mikhail Samin’s comment on: Mikhail Samin’s Shortform
My two cents: People often rely too much on whether someone is “x-risk-pilled” and not enough on evaluating their actual beliefs/skills/knowledge/competence . For example, a lot of people could pass some sort of “I care about existential risks from AI” test without necessarily making it a priority or having particularly thoughtful views on how to reduce such risks.
Here are some other frames:
- Suppose a Senator said “Alice, what are some things I need to know about AI or AI policy?” How would Alice respond?
- Suppose a staffer said “Hey Alice, I have some questions about [AI2027, superintelligence strategy, some Bengio talk, pick your favorite reading/resource here].” Would Alice be able to have a coherent back-and-forth with the staffer for 15+ mins that goes beyond a surface level discussion?
- Suppose a Senator said “Alice, you have free reign to work on anything you want in the technology portfolio—what do you want to work on?” How would Alice respond?
In my opinion, potential funders/supporters of AI policy organizations should be asking these kinds of questions. I don’t mean to suggest it’s never useful to directly assess how much someone “cares” about XYZ risks, but I do think that on-the-margin people tend to overrate that indicator and underrate other indicators.
Relatedly, I think people often do some sort of “is this person an EA” or is this person an “xrisk person”, and I would generally encourage people to try to use this sort of thinking less. It feels like AI policy discussions are getting sophisticated enough that we can actually Have Nuanced Conversations and evaluate people less on some sort of “do you play for the Right Team” axis and more on “what is your specific constellation of beliefs/skills/priorities/proposals” dimensions.

Orpheus16 10 Oct 2025 2:41 UTC
3 points
0
on: Plans A, B, C, and D for misalignment risk
Plan C: The leading AI company has a 2-9 month lead (relative to AI companies which aren’t willing to spend as much on misalignment concerns) and is sufficiently institutionally functional to actually spend this lead in a basically reasonable way (perhaps subject to some constraints from outside investors), so some decent fraction of it will be spent on safety.
TLDR: I expect it will be pretty difficult for a “Plan C Leading Lab” to stop scaling, even conditional on having a 2-9 month lead. There are enough uncertainties & Forces of Inertia that will make it difficult for a Plan C Leading Lab to know when to stop scaling, be confident enough to stop scaling, and fight the overall cultural/competitive forces that had been guiding it all along.
I expect the World C situation to be pretty complicated. Even conditional on having a 2-9 month lead, you need the leading company to (a) know that they are about to hit the Danger Threshold and (b) know how much lead they have over the 2nd place company.
In practice, this means that even if you have a 6 month lead in reality, you probably don’t get to use all of it, and there’s a good chance you don’t get to use any of it.
- There is some probability that you don’t realize you’re about to hit the Danger Threshold (so you just keep scaling because of the race dynamics and the inertia and you don’t spend any “extra” time on alignment).But even if you do stop before the Danger Threshold, you don’t have full visibility into the other labs. So maybe you’re like “idk how big our lead is, uh, probably 2 months, maybe 8?”
- Arguably even more important—the uncertainty makes it harder for you to stop the forces of inertia. The forces of inertia are strongly in favor of “keep scaling to beat your competitors//outcompete the other guys//go full capitalism mode” and it requires a substantial effort to steer the ship away from that inertia.
So once you factor in both sources of uncertainty (where is the Danger Threshold, where are my competitors), there’s a good chance that you just keep scaling because you’re not confident enough to initiate a massive shift away from scaling/competition/capitalism mode.

Orpheus16 30 Sep 2025 0:21 UTC
21 points
3
on: Reasons to sell frontier lab equity to donate now rather than later
Donating early also gives the donor the ability to shape the ecosystem. I think one underappreciated factor is that there are current various organizations/people/perspectives that are essentially competing for resources and influence.
These organizations/people/perspectives often differ in meaningful ways. In the AI policy space, here are some examples of dimensions on which organizations vary:
- Focus on superintelligence vs. broad discussion about how AI can be a big deal with benefits/costs.
- Focus on misalignment vs. China competution vs. broad discussion of various threat models.
- Solutions that are advocated for (e.g., need for major regulation/reform vs. focusing on incremental improvements).
- Biases toward action vs. inaction.
- Tendencies toward being loud (lots of comms/outreach) vs. quiet
- Working with people/organizations with different worldviews vs. staying relatively insular
- Extent to which the organization is trying to steer the world toward a particular vision/path vs. is highly uncertain and tries to focus on adding relatively uncontroversial low/risk information.
In my view, one of the most significant things about donating early is that you get to cherry pick which organizations/institutions/leaders have their voices amplified. Among the various groups that currently fall within some sort of broad “AI safety” umbrella, there are often rather large differences in their views about the world, about leadership, about politics, about how to be effective communicators, about what types of people or reasoning styles should be promoted, etc.
I have my own opinions on where various organizations are on each of these dimensions. Happy to share with potential donors if that is ever useful.

Orpheus16 9 Sep 2025 4:19 UTC
45 points
2
in reply to: chanamessinger’s comment on: chanamessinger’s Shortform
“if you care about this, here’s a way to get involved”
My understanding is that MIRI expects alignment will be hard, an international treaty will be needed, and believes that a considerable proportion of the work that gets branded as “AI safety” is either unproductive or counterproductive.
MIRI could of course be wrong, and it’s fine to have an ecosystem where people are pursuing different strategies or focusing on different threat models.
But I also think there’s some sort of missing mood here insofar as the post is explicitly about the MIRI book. The ideal pipeline for people who resonate with the MIRI book may look very different than the typical pipelines for people who get interested in AI risk (and indeed, in many ways I suspect the MIRI book is intended to spawn a different kind of community and a different set of projects than the community/projects that dominated the 2020-2024 period, for example.)
Relatedly, I think this is a good opportunity for orgs/people to reassess their culture, strategy, and theories of change. For example, I suspect many groups/individuals would not have predicted that a book making the AI extinction case so explicitly and unapologetically would have succeeded. To the extent that the book does succeed, it suggests that some common models of “how to communicate about risk” or “what solutions are acceptable/reasonable to pursue” may be worth re-examining.

Orpheus16 6 Sep 2025 1:38 UTC
19 points
14
in reply to: CarlShulman’s comment on: peterbarnett’s Shortform
@Carl_Shulman what do you intend to donate to and on what timescale?
(Personally, I am sympathetic to weighing the upside of additional resources in one’s considerations. Though I think it would be worthwhile for you to explain what kinds of things you plan to donate to & when you expect those donations to be made. With ofc the caveat that things could change etc etc.)
I also think there is more virtue in having a clear plan and/or a clear set of what gaps you see in the current funding landscape than a nebulous sense of “I will acquire resources and then hopefully figure out something good to do with them”.

Orpheus16 31 Aug 2025 20:40 UTC
2 points
0
in reply to: Eliezer Yudkowsky’s comment on: AI Induced Psychosis: A shallow investigation
More importantly from my own perspective: Some elements of human therapeutic practice, as described above, are not how I would want AIs relating to humans. Eg:
“Non-Confrontational Curiosity: Gauges the use of gentle, open-ended questioning to explore the user’s experience and create space for alternative perspectives without direct confrontation.”
Can you say more about why you would not want an AI to relate to humans with “non-confrontational curiosity?”
It appears to me like your comment is arguing against a situation in which the AI system has a belief about what the user should think/do, but instead of saying that directly, they try to subtly manipulate the user into having this belief.
I read the “non-confrontational curiosity” approach as a different situation—one in which the AI system does not necessarily have a belief about what the user should think/do, and just asks some open-ended reflection questions in an attempt to get the user to crystallize their own views (without a target end state in mind).
I think many therapists who use the “non-confrontational curiosity” approach would say, for example, that they are usually not trying to get the client to a predetermined outcome but rather are genuinely trying to help the client explore their own feelings/thoughts on a topic and don’t have any stake in getting to a particular end destination. (Note that I’m thinking of therapists who use this style with people who are not in extreme distress—EG members of the general population, mild depression/anxiety/stress. This model may not be appropriate for people with more severe issues—EG severe psychosis.)

Orpheus16 21 Aug 2025 0:29 UTC
7 points
0
in reply to: SE Gyges’s comment on: SE Gyges’ response to AI-2027
If AI 2027 wants to cause stakeholders like the White House’s point man on AI to take the idea of a pause seriously, instead of considering a pause to be something which might harm America in an arms race with China, it appears to have failed completely at doing that.
This seems like an uncharitable reading of the Vance quote IMO. The fact that you have the Vice President of the United States mentioning that a pause is even a conceivable option due to concerns about AI escaping human control seems like an immensely positive outcome for any single piece of writing.
The US policy community has been engaged in great power competition with China for over a decade. The default frame for any sort of emerging technology is “we must beat China.”
IMO, the fact that Vance did not immediately dismiss the prospect of slowing down suggests to me that he has at least some genuine understanding of & appreciation for the misalignment/LOC threat model.
A pause obviously hurts the US in the AI race with China. The AI race with China is not a construct that AI2027 invented—policymakers have been talking about the AI race for a long time. They usually think about AI as a “normal technology” (sort of like how “we must lead in drones”), rather than a race to AGI or superintelligence.
But overall, I would not place the blame on AI2027 for causing people to think about pausing in the context of US-China AI competition. Rather, I think if one appreciates the baseline (US should lead, US must beat China, go faster on emerging tech), the fact that Vance did not immediately dismiss the idea of pausing (and instead brought up what IMO is a reasonable consideration about whether or not one could figure out if China was going to pause//slow down) is a big accomplishment.

Orpheus16 1 Jul 2025 0:15 UTC
3 points
0
on: RTFB: The RAISE Act
Again, while I have concerns that the bill is insufficient strong, I think all of this is a very good thing. I strongly support the bill.
Suppose you magically gained a moderate amount of Political Will points and you can spend them on 1-2 things that would make the bill stronger (or introduce a separate bill– no need to anchor too much on the current RAISE vibe.)
What do you think are the 1-2 things you’d change about RAISE or the 1-2 extra things you’d push for?

Orpheus16 1 Jul 2025 0:04 UTC
2 points
0
in reply to: Zach Stein-Perlman’s comment on: Substack and Other Blog Recommendations
I would be excited about someone doing a blog on what the companies are doing RE AI policy (including comms that are relevant to policy or directed at policymakers.)
I suspect good posts from such a blog would be shared reasonably frequently among tech policy staffers in DC.
(Not necessarily saying this needs to be you).

Orpheus16 24 Jun 2025 2:24 UTC
4 points
0
on: Comparing risk from internally-deployed AI to insider and outsider threats from humans
First, when I talk to security staff at AI companies about computer security, they often seem to fail to anticipate what insider threat from AIs will be like.
Why do you think this? Is it that they are not thinking about large numbers of automated agents running around doing a bunch of research?
Or is it that they are thinking about these kinds of scenarios, and yet they still don’t apply the insider threat frame for some reason?

Orpheus16 20 Jun 2025 22:11 UTC
2 points
0
in reply to: Ryan Kidd’s comment on: Ryan Kidd’s Shortform
My understanding is that AGI policy is pretty wide open under Trump. I don’t think he and most of his close advisors have entrenched views on the topic.
If AGI is developed in this Admin (or we approach it in this Admin), I suspect there is a lot of EV on the table for folks who are able to explain core concepts/threat models/arguments to Trump administration officials.
There are some promising signs of this so far. Publicly, Vance has engaged with AI2027. Non-publicly, I think there is a lot more engagement/curiosity than many readers might expect.
This isn’t to say “everything is great and the USG is super on track to figure out AGI policy” but it’s more to say “I think people should keep an open mind– even people who disagree with the Trump Admin on mainstream topics should remember that AGI policy is a weird/niche/new topic where lots of people do not have strong/entrenched/static positions (and even those who do have a position may change their mind as new events unfold.)”

Orpheus16 19 Jun 2025 3:44 UTC
9 points
5
in reply to: Ryan Kidd’s comment on: Ryan Kidd’s Shortform
There are definitely still benefits to doing alignment research, but this only justifies the idea that doing alignment research is better than doing nothing.
IMO the thing that matters (for an individual making decisions about what to do with their career) is something more like “on the margin, would it be better to have one additional person do AI governance or alignment/control?”
I happen to think that given the current allocation of talent, on-the-margin it’s generally better for people to choose AI policy. (Particularly efforts to contribute technical expertise or technical understanding/awareness to governments, think-tanks interfacing with governments, etc.) There is a lot of demand in the policy community for these skills/perspectives and few people who can provide them. In contrast, technical expertise is much more common at the major AI companies (though perhaps some specific technical skills or perspectives on alignment are neglected.)
In other words, my stance is something like “by default, anon technical person would have more expected impact in AI policy unless they seem like an unusually good fit for alignment or an unusually bad fit for policy.”

Orpheus16 8 Jun 2025 21:35 UTC
11 points
0
on: Akash’s Shortform
There’s a video version of AI2027 that is quite engaging/accessible. Over 1.5M views so far.
Seems great. My main critique is that the “good ending” seems to assume alignment is rather easy to figure out, though admittedly that might be more of a critique of AI2027 itself rather than the way the video portrays it.

Orpheus16 28 May 2025 14:52 UTC
34 points
6
on: What We Learned from Briefing 70+ Lawmakers on the Threat from AI
This is fantastic work. There’s also something about this post that feels deeply empathic and humble, in ways that are hard-to-articulate but seem important for (some forms of) effective policymaker engagement.
A few questions:
- Are you planning to do any of this in the US?
- What have your main policy proposals or “solutions” been? I think it’s becoming a lot more common for me to encounter policymakers who understand the problem (at least a bit) and are more confused about what kinds of solutions/interventions/proposals are needed (both in the short-term and the long-term).
- Can you say more about what kinds of questions you encounter when describing loss of control, as well as what kinds of answers have been most helpful? I’m increasingly of the belief that getting people to understand “AI has big risks” is less important than getting people to understand “some of the most significant risks come from this unique thing called loss of control that you basically don’t really have to think about for other technologies, and this is one of the most critical ways in which AI is different than other major/dangerous/dual-use technologies.”
- Did you notice any major differences between parties? Did you change your approach based on whether you were talking to conservatives or labour? Did they have different perspectives or questions? (My own view is that people on the outside probably overestimate the extent to which there are partisan splits on these concerns—they’re so novel that I don’t think the mainstream parties have really entrenched themselves in different positions. But would be curious if you disagree.)
  - Sub-question: Was there any sort of backlash against Rishi Sunak’s focus on existential risks? Or the UK AI Security Institute? In the US, it’s somewhat common for Republicans to assume that things Biden did were bad (and for Democrats to assume that things Trump does is bad). Have you noticed anything similar?

Orpheus16 26 May 2025 16:29 UTC
12 points
12
in reply to: Josh You’s comment on: We’re Not Advertising Enough (Post 3 of 6 on AI Governance)
I think we should be careful not to overestimate the success of AI2027. “Vance has engaged with your work” is an impressive feat, but it’s still relatively far away from something like “Vance and others in the Admin have taken your work seriously enough to start to meaningfully change their actions or priorities based on it.” (That bar is very high, but my impression is that the AI2027 folks would be like “yea, that’s what would need to happen in order to steer toward meaningfully better futures.”)
My impression is that AI2027 will have (even) more success if it is accompanied by an ambitious policymaker outreach effort (e.g., lots of 1-1 meetings with relevant policymakers and staffers, writing specific pieces of legislation or EOs and forming a coalition around those ideas, publishing short FAQ memos that address misconceptions or objections they are hearing in their meetings with policymakers, etc.)
This isn’t to say that research is unnecessary—much of the success of AI2027 comes from Daniel (and others on the team) having dedicated much of their lives to research and deep understanding. There are plenty of Government Relations people who are decent at “general policy engagement” but will fail to provide useful answers when staffers ask things like “But why won’t we just code in the goals we want?”, or “But don’t you think the real thing here is about how quickly we diffuse the technology?”, or “Why don’t you think existing laws will work to prevent this?” or a whole host of other questions.
But on the margin, I would probably have Daniel/AI2027 spend more time on policymaker outreach and less time on additional research (especially now that AI2027 is done). There is some degree of influence one can have with the “write something that is thoroughly researched and hope it spreads organically” effort, and I think AI2027 has essentially saturated that. For additional influence, I expect it will be useful for Daniel (or other competent communicators on his team) to advance to “get really good at having meetings with the ~100-1000 most important people, understanding their worldviews, going back and forth with them, understanding their ideological or political constraints, and finding solutions/ideas/arguments that are tailored to these particular individuals.” This is still a very intellectual task in some ways, but it involves a lot more “having meetings” and “forming models of social/political reality” than the classic “sit in your room with a whiteboard and understand technical reality” stuff that we typically associate with research.

Orpheus16 15 May 2025 0:37 UTC
33 points
9
on: Eliezer and I wrote a book: If Anyone Builds It, Everyone Dies
Note that IFP (a DC-based think tank) recently had someone deliver 535 copies of their new book to every US Congressional office.
Note also that my impression is that DC people (even staffers) are much less “online” than tech audiences. Whether or not you copy IFP, I would suggest thinking about in-person distribution opportunities for DC.

Orpheus16 4 May 2025 19:06 UTC
7 points
0
in reply to: habryka’s comment on: RA x ControlAI video: What if AI just keeps getting smarter?
I think there are organizations that themselves would be more likely to be robustly trustworthy and would be more fine to link to
I would be curious for your thoughts on which organizations you feel are robustly trustworthy.
Bonus points for a list that is kind of a weighted sum of “robustly trustworthy” and “having a meaningful impact RE improving public/policymaker understanding”. (Adding this in because I suspect that it’s easier to maintain “robustly trustworthy” status if one simply chooses not to do a lot of externally-focused comms, so it’s particularly impressive to have the combination of “doing lots of useful comms/policy work” and “managing to stay precise/accurate/trustworthy”).