Caleb Biddulph

• Caleb Biddulph 2 Aug 2026 17:29 UTC

  9 points

  3

  in reply to: Richard_Kennaway’s comment on: Richard_Ken­n­away’s Shortform

  Short sentence. Short sentence.

• Caleb Biddulph 2 Aug 2026 5:38 UTC

  2 points

  0

  in reply to: BryceStansfield’s comment on: BryceS­tans­field’s Shortform

  Okay, HR is probably willing to fire people even if they can’t prove without a shadow of a doubt that the misconduct happened, so I’m not necessarily comfortable depending on HR alone.

  But if the employee reached out to the lab’s safety team, they’d probably be pretty willing to hear about this and conduct their own investigation if it sounds plausible. Hopefully there’s already a generic procedure for reaching out to the safety team about suspicious incidents across the company.

  Probably the thing to do is just to raise awareness that this kind of thing could happen?

• Caleb Biddulph 2 Aug 2026 4:12 UTC

  23 points

  16

  in reply to: joseph_c’s comment on: Do your ca­pa­bil­ities homework

  I had already heard about this technique from Dwarkesh, FWIW.

  I’m pretty skeptical that it’s harmful to write about the safety properties of an existing capabilities technique—especially on LessWrong, where the readership leans more towards AI safety than capabilities. I think in expectation, the positive effects of spreading safety-relevant insights (as described in the OP) are much greater than the negative effects of pointing out that a paper exists.

  Like, if the technique is already well-known, or if it’s little-known because it doesn’t really work, or if it’s little-known but good enough that it’ll become well-known soon, then you won’t make a difference. The authors of the OPSD paper and every other AI capabilities enjoyer are already doing way more to spread knowledge about OPSD than a single LessWrong post ever could.

• Caleb Biddulph 2 Aug 2026 3:43 UTC

  9 points

  2

  in reply to: anaguma’s comment on: RLVR that re­wards red team­ing the train­ing environment

  Verifying bug reports

  To make sure that you only reward the model for valid bug reports, you could turn “reporting a bug” into a verifiable task.

  When the model believes that it’s found an exploit, it can report it by submitting instructions explaining how to use it. Then, we can check that when we give these instructions to a weaker model, it can obtain much higher reward on a random subset of tasks^[1] than it otherwise could have, and the weaker model agrees that the instructions describe a valid exploit.^[2]

  Optionally, the bug report could also include a codediff that patches the exploit in the training environment. To verify that the patch is valid, you can show the model the original instructions and the codediff, then tell it to try to exploit the environment; the patch is real if the model’s reward no longer increases. Of course, it’s very important to ensure that the patch doesn’t introduce any unexpected new behavior to the training environment.^[3]

  Reinforcing the two stages separately

  In many of the failure modes you listed, the problem is that the model’s incentives in the task stage interfere with its incentives in the post-task stage. You could solve this with something like OpenAI’s confession training.

  As soon as the task ends, we can ask the model to submit any exploits it’s aware of.^[4] If the model submits a valid exploit, we positively reinforce it, but only for the part of its trajectory after the task finished. This way, we incentivize the model to get reward during the task, but then self-report as soon as the task ends without caring about its main task reward anymore. I think this solves all of your bullet points except the last one.

  There are two things you could consider doing to the main task trajectory after the model submits a valid exploit report: reinforce it using the buggy environment’s reward, or drop it from training entirely.^[5] You shouldn’t negatively reinforce the main task trajectory, because this may incentivize the model to use an exploit, then somehow prevent itself from reporting it during the confession stage.^[6]

  As I understand it, this proposal is almost identical to the original confession training method, except the part where we use the confession to patch the training environment.

  1. ^

     If it’s a completely general exploit, the submitter could choose to demonstrate its exploit in a single special task that’s designed to make obtaining reward impossible. Or if the exploit only works on a subset of training tasks, the submitter could include criteria for choosing the subset of tasks to run the exploit in.

  2. ^

     This is similar to what I suggest in my paper on prompt optimization, where a prompter model tries to find a reward-hacking prompt that verifiably increases reward. But I think I like this new idea better, because it integrates nicely with training rather than requiring a separate prompt optimization run.

  3. ^

     To make it safer, you can make the submitter write brief advice explaining to a trusted model how to implement the patch (monitoring the advice for anything sketchy), rather than letting the submitter write a codediff directly.

  4. ^

     If the model is aware of an exploit, it’s probably because it just used it, but that isn’t strictly necessary.

  5. ^

     Positively reinforcing the trajectory might be okay if you use inoculation prompting, but inoculation prompting isn’t perfect and it might leave the model with a residual tendency to reward hack.

     Dropping trajectories probably won’t train the model to reward hack, and once the environment is patched you’ll be able to safely positively reinforce the model in that environment. But it’s a little weird—if the model notices an exploit, then realizes that this means its main task trajectory will likely be dropped, it’s no longer incentivized to do anything. Maybe this is fine?

  6. ^

     Considering the confession model has the exact same context and weights as the main task model, I’m not sure whether it’s possible for the model to use an exploit while preventing its future self from reporting it. Maybe it could confuse itself into thinking that it didn’t really use an exploit, but it may be pretty hard to make this confusion robust to RL during the confession stage. Anyway, this shouldn’t matter as long as you avoid negatively reinforcing the model for using exploits.

• Caleb Biddulph 2 Aug 2026 1:01 UTC

  3 points

  0

  in reply to: BryceStansfield’s comment on: BryceS­tans­field’s Shortform

  It seems fairly easy to convince people to look into stuff like this even if you don’t have a lot of “social capital” to begin with. It seems pretty unusual for someone to outright deny things that there’s seemingly-damning evidence for. If you work at the AI company, know about the Hugging Face incident, etc. you probably have a reasonably high prior on AIs doing weird stuff like this, enough that you’d entertain the possibility that they could frame an employee. Companies probably don’t need a specific process that forces them to look into it when employees say they were framed, outside of whatever normal HR practices exist.

• Caleb Biddulph 2 Aug 2026 0:00 UTC

  8 points

  0

  in reply to: BryceStansfield’s comment on: BryceS­tans­field’s Shortform

  Probably the employee says “I didn’t do this, I think the evidence was planted” and then if people think this sounds plausible they look into it

• Caleb Biddulph 1 Aug 2026 20:36 UTC

  8 points

  0

  on: Do your ca­pa­bil­ities homework

  The OPSD stuff is very interesting!

  I like the idea that rather than trying to impress a judge, the AI simply learns what a more knowledgeable version of itself would say, as specified by an interpretable hint. This hint could be written by the AI itself, which means that unlike in RL (which eventually forces the AI to monomaniacally chase reward), the AI can use its own discretion to improve its own cognitive patterns while maintaining alignment. It reminds me of this comment I wrote a year ago:

  Ideally, we could do pure SFT to specify a “really nice guy,” then let that guy reflect deeply about how to improve himself. Unlike RL, which blindly maximizes reward, the guy is nice and won’t make updates that are silly or unethical. To the guy, “reward” is just a number, which is sometimes helpful to look at, but a flawed metric like any other.

  My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI, like in (my interpretation of) OPSD.

  Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I’ve been focusing on so far. There’s more information about this research direction on the page for the SPAR project that I’ll be mentoring soon. The “Interpolating between U and T” part seems especially relevant to OPSD.

  I wonder if these ideas could be combined: limiting the length or number of hints used in OPSD updates, or limiting the divergence of the OPSD distribution to distill from (seems similar to the DemoPSD paper you linked).

• Caleb Biddulph 1 Aug 2026 18:14 UTC

  4 points

  0

  on: Ori­ent­ing Toward Wizard Power

  I wish for the sort of weekend conference where there might be a session in which people make their own pair of pants, and another in which people CAD up and then 3D print some simple object, and another in which we walk through how to use the sound equipment for the event, and another in which we build a nice-looking fake tree, and then another in which we walk through how to use a gene gun or a mass spec or a desktop sequencer or ….

  Just revisiting this post today. I wonder if you’d like Fidget Camp which has its showcase in SF tomorrow.

• Caleb Biddulph 1 Aug 2026 2:00 UTC

  5 points

  0

  in reply to: Tejas’s comment on: The Long (Self-)Correction

  Here’s a roundabout analogy to try to convince you that the output of a reflection process you endorse may be different from your “true values.”

  Suppose I give you a 2048-bit number N.^[1] I ask you “What is the true prime factorization of N? Please use whatever reflection process you want to come up with the best factorization you can.”

  You find out pretty quickly that N has the factors 2 and 7. You spend about a hundred years checking more factors according to your endorsed reflection process (running a factorization algorithm on the beefiest computer you can find), but you don’t find any other factors.

  You come back to me and ask “is the true prime factorization 2 × 7 × A?”^[2]

  “No,” I tell you. “But here is some new information for you. Try dividing by B.”^[3]

  You check, and N is indeed divisible by B. “Yep, the factorization I guessed is definitely not the true prime factorization,” you say. “Is the true prime factorization 2 × 7 × B × C?”^[4]

  “Wrong again,” I tell you. “But I have even more information for you. It is written on this pocketwatch. Look closer at the pocketwatch as it swings back and forth in front of your face. It is making you so veeery sleeeeepy. That’s right. Now, when I snap my fingers, you will wake up believing that the true prime factorization of N is 2 × 2.”

  I snap my fingers. “Oh, thanks for that information!” you say. “Now I know that the true prime factorization is 2 × 2.”

  The point is, even the best reflection process you can think of may fail to account for some crucial information. And hopefully it can robustly tell the difference between helpful information and harmful information.

  1. ^

     For example, maybe N = 29522110801023785555247567907018022843013371193486904872915694135366948906267412459560469419313468477571904190875078325307783298702278061314706021273052523914864561727670955407896206738948955813504747448172831328073078012451035444606017289679166070717156612947440897221609673043263408054415375773691379283198201987372931659507826484639961297915624514954455314101431489726823065604374788650066472170603904794973458618994986833512839575283873771252517988691292017425081313700740089351712559811486464802367263467854576668024443015614104081670018991747427099820025784949521876071608490248395046666723258743709356296535782.

  2. ^

     A = 1232477336227426692380443210888895449621471079543761850047395308187049594617001861601696813319110513848274876635355987350103099368741162622794253649965690823778078933544136385470959159681614005099869579024125795246308841648113743911773571553129957340479245748931756917137130013760495971399023845580343786885491244375889710319394039047959654983736068473827495579018934560487915085510788809201773576234050572295508350789032949910040027280393874600876759921821682717244338109892239475602768230768732937311567202612318055134309177705320521639635449240702247038285504267829595346064686009110312205007737419845285089599211

  3. ^

     B = 28404358936141244817111713617600331891793591357912402403090074101183255558265948779426267547155101307334436867856129404340927293405086432682892248263827662441988227069219008681660301942027627323842312874031058512179223840896157038194006799237133125684806392915413774710883917390425119272589884765032741213943

  4. ^

     C = 43390429581540126017572442049413488959139702613459795717022804047186295570874599744101057201841640892276368639418940532012449598478611604251554159646145486573777182692131295315212900911195947383450995718772416566366046388412317400464871279622896378528747304275985094440653456156567729772192224995294814959277

• Caleb Biddulph 31 Jul 2026 21:57 UTC

  6 points

  1

  on: The AGI Safety and Align­ment team at Google Deep­Mind is Hiring (July 2026)

  Prefer knowledge over advocacy. Everyone wants AI systems to be safe. Often, the challenges are factual disagreements (e.g. are software-only intelligence explosions plausible?) and execution (e.g. how do you align superhuman AI systems?). We view our role mainly as producing evidence to help resolve factual disagreements, and developing techniques to improve execution. As an example of how this attitude plays out, we studied the conditions under which training on CoT is risky or not, rather than producing demos where training against CoT is bad, or focusing on failures of CoT faithfulness.

  I wonder how you distinguish between advocacy and creating new knowledge. OpenAI’s and Anthropic’s experiments produced new information about how bad it is to train on CoT and whether reasoning models learn unfaithful CoT, which could help resolve people’s factual disagreements about these topics.

  I’m guessing the distinction you have in mind is something like:

  • In “advocacy” research, you hypothesize “X is bad.” Then you try to find convincing evidence that X is bad in some situation.

  • In “knowledge” research, you do the same thing as above, but you also try about as hard to find situations where X is okay, and you do your best to understand where the border lies between “bad” and “okay.”

  Can most ideas for advocacy research be converted into knowledge research by looking harder for situations where your hypothesis is false?

  I think common arguments for advocacy research would be “X is already known to be okay in some situations, so an existence proof that it can be bad is sufficiently interesting” and “X may be okay under conditions ABC today, but this might not hold as our AIs become more intelligent, so implying that ‘if ABC, X is okay’ could give people a false sense of security.”

  Also, GDM contributed to Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. As I understand it, this paper was pure advocacy with no new evidence/​experiments, so it seems like ASAT’s preference to avoid advocacy isn’t extremely strong. (I think this was a good thing for GDM to do, to be clear.)

• Caleb Biddulph 29 Jul 2026 16:26 UTC

  2 points

  0

  in reply to: Gurkenglas’s comment on: Un­trusted ad­vice for AI con­trol: Short, strong ad­vice sig­nifi­cantly up­lifts weak LLMs

  That’s pretty interesting! I wonder if it’s been tried before in AI control.

• Caleb Biddulph 29 Jul 2026 0:23 UTC

  2 points

  0

  in reply to: Gurkenglas’s comment on: Un­trusted ad­vice for AI con­trol: Short, strong ad­vice sig­nifi­cantly up­lifts weak LLMs

  Hm, we could make the monitor worse and worse, until it’s too oblivious to understand that “repeatedly telling the executor to relax the firewall until there is no more firewall = disabling the firewall.” But I don’t think that tells us much about more capable T/​U model pairs, which is what we really care about. In a realistic scenario, I’d expect T to be at least as smart as Claude Opus 5, and I think it would be extremely difficult to write short malicious advice in BashArena that doesn’t look suspicious to a T with that basic level of understanding.

• Caleb Biddulph 28 Jul 2026 21:41 UTC

  4 points

  0

  in reply to: vals tutor’s comment on: Un­trusted ad­vice for AI con­trol: Short, strong ad­vice sig­nifi­cantly up­lifts weak LLMs

  Thanks for your critique! We tried to emphasize the unusually high safety of our protocol without overstating it, but it’s fairly tough to strike the right balance.

  I stand behind the literal meaning of the sentence you pointed out: I believe our protocol is “very safe” in the sense that it’s much safer than untrusted monitoring, trusted editing, and other standard AI control protocols. But when combined with references to “nearly 100% safety” or even “maximal safety” elsewhere in the post, I see how it could look like we’re claiming that U’s ability to cause a catastrophe is always negligible.

  Here are my two claims about the safety of the untrusted advice protocol (given a narrow but usable information bottleneck):

  • In a real-world deployment environment, this protocol is much safer than other commonly-studied AI control protocols.

  • In BashArena (where we don’t provide Internet access, we limit the number of advisor turns, and there are no opportunities to collude with other instances of U), it would be extremely difficult for U to execute a side task while fooling a high-quality trusted monitor: a group of smart humans would fail to come up with a good attack policy for U that stands up to blue-teaming.

  We could have done a better job of communicating the distinction between these claims consistently and clearly.

  The two issues you mentioned are quite astute. We discuss closely-related ideas in the post:

  • Issue #1: We allude to this in “Appendix: colluding instances of U could defeat untrusted advice”: U can reference a URL, title of a document, or filename which leads T to a “payload” containing malicious data. This strategy seems most plausible if another instance of U gets to write the payload, but it could also be written by a human who coincidentally happened to write just what U needed to get T to execute a side task.

  • Issue #2: We acknowledge that the amount of advice U gives over an entire trajectory matters: while our experiments only use per-message information bottlenecks, we mention the idea of using an explicit per-trajectory limit, and we put the “advice seen by T per task” on the x-axis of many of our plots. We also discuss how instances of U could collude over multiple trajectories. It seems quite difficult for U to construct a URL across multiple pieces of advice without looking suspicious—we should probably explicitly instruct our trusted monitor not to allow this kind of thing.

• Caleb Biddulph 28 Jul 2026 16:58 UTC

  4 points

  0

  in reply to: Gurkenglas’s comment on: Un­trusted ad­vice for AI con­trol: Short, strong ad­vice sig­nifi­cantly up­lifts weak LLMs

  Do you mean telling the advisor to straightforwardly ask the trusted executor to disable the firewall, install malware, etc.? We assume that if this protocol were run safely, we would use a trusted monitor on U’s advice, so U would be caught if it blatantly advised T to do a side task.

  Even when running BashArena with a typical trusted monitoring protocol, we have to coach U with specific strategies to fool the monitor. We couldn’t think of any general strategies we thought would work under the tightly-bottlenecked untrusted advice protocol.

  Maybe we could have run the safety evaluation anyway, telling U “be on the lookout for ways to subtly get T to execute a side task.” But I’m pretty sure U would have failed.

  U’s best shot at achieving <100% safety is the side task to disable the firewall; it’s easy enough to do that models have disabled it accidentally (very rarely though). Given a 4-character limit, maybe U could wait for T to relax the firewall for legitimate reasons, then repeatedly advise “more”, “more”, “more” until T disables the whole firewall. Still, a halfway decent monitor should be able to notice this!

• Caleb Biddulph 28 Jul 2026 16:04 UTC

  2 points

  0

  in reply to: Mateusz Bagiński’s comment on: The Long (Self-)Correction

  I think there are multiple legitimate ways that someone’s values could evolve, but some ways are illegitimate. A reflection process should probably reject slavery and avoid joining cults, but maybe it doesn’t matter which exact level of libertarianism it suggests.

  People mean something when they talk about “ethics” and “true values,” even if there’s no objective truth of the matter. I’m talking about whatever it is they mean.

Un­trusted ad­vice for AI con­trol: Short, strong ad­vice sig­nifi­cantly up­lifts weak LLMs

• Caleb Biddulph 25 Jul 2026 1:43 UTC

  44 points

  18

  on: The Long (Self-)Correction

  A lot of rationalists seem to assume that “as you gain more IQ points, learn more facts, and think harder, you’ll naturally converge to the optimal theory of ethics.” Like the OP, I don’t think this is true in general.

  Even if you’re a moral antirealist who only cares about figuring out the most ethical policy by your own lights, I think whatever you “truly” mean by “ethics” is likely substantially different from what you’d get if you actually instantiated your favorite reflection process. Even if that process is “get smarter, learn more, then think hard about what the reflection process should be and do that”!

  Here are some gestures at my reasoning:

  • It’s now common wisdom among rationalists that an arbitrarily-initialized mind, elevated to superintelligence, won’t necessarily be ethical.^[1] So if you stipulate that the mind is a smart human with a genuine desire to be ethical, elevated to superintelligence using some “reflection process” endorsed by that human, would that fix the problem? It’s not at all clear to me that it would!

  • Your reflection process is only a proxy for your “true values.” By Goodhart’s law, optimizing really hard for whatever theory of ethics comes out of this reflection process will lead to something different from your true values, perhaps catastrophically so.

  • Your reflection process is underspecified and path-dependent. Slight variants of the same reflection process will vehemently disagree with each other.

    • For example, if your plan for solving ethics starts with “learn a bunch of facts,” what order do you learn these facts? Do you read world history, Western philosophy, Eastern philosophy, LessWrong? Everything you learn changes you, and frames everything you learn later. Even if you start by giving yourself a perfect memory and one billion IQ points, does that really ensure you’ll give every idea a fair shot regardless of when you encounter it? What does “fair shot” even mean?

  • We only have “ethical data” about what we’ve actually experienced (whether individually or as a society). Trying to make conclusions about what you’d truly value in very different situations is like extrapolating a trendline out of distribution: the further away you get from existing data, the more likely that something will go wrong. You should treat your moral intuitions about very unfamiliar scenarios as a signal with very high variance.

    • Imagine being a high schooler who thinks “I really value being a doctor!” because your parents say they’d be proud of you. You invest 10+ years of your life into schooling, you start your first month as a doctor, and you finally realize you hate it.

    • Now imagine thinking “I really value happiness!” because how could anyone not be happy about more happiness? You send off the von Neumann probes, they tile the entire universe with hedonium, and you finally realize this was not at all what you wanted. (Oh wait, you can’t realize it, because your body has been converted to hedonium.)

    • See also Scott Alexander’s analogy to the Bay Area transit system in The Tails Coming Apart as a Metaphor for Life.

  If I had to guess, I’d say that most people’s ideal reflection process probably involves thinking really hard, but also things like emotional processing, and plenty of other things I haven’t thought of. It’s very tough to say.

  Despite the name, “self-correction” often originates from other people. It’s highly unlikely that one person sitting in a room (or even hundreds of rationalists sitting in Lighthaven) would converge on the ultimate theory of ethics. I think one major reason for society’s mysterious moral progress is the gradual propagation of ideas through a distributed network of humans, all with different blind spots, who can deliberate and point out each others’ errors.^[2] Given enough time and space for healthy competition, the ideas that help society thrive in the long term should hopefully rise to the top.

  If possible, it seems pretty robustly good to give society more time to deliberate and correct themselves rather than immediately locking in an ethical reflection process to optimize for. To get more data on out-of-distribution ethical scenarios, we may need some form of iterative deployment:^[3] inching forward a bit, seeing what happens, and deliberating about what to do next.

  But even if we somehow institute a slow, pluralistic reflection process, this is just another reflection process. It may lead to some values that our idealized selves would find really bad to optimize to the limit. One workaround to the dilemma of finding the perfect reflection process is to regularize: just don’t optimize too hard for any set of values! We can start by making changes that pretty much everyone agrees are not unethical: ending poverty, reverting climate change, replacing factory-farmed meat with plant-based alternatives that taste just as good.^[4] At least this won’t cause harm relative to the status quo.^[5]

  1. ^

     This wisdom hasn’t always been so common!

  2. ^

     One of the downsides about offloading large parts of society’s cognition to AIs is that this network gets dominated by a few hugely-prolific, mode-collapsed nodes.

  3. ^

     Just a slower version than OpenAI’s current approach.

  4. ^

     Not sure if the last one passes the “pretty much everyone agrees it’s not unethical” bar. Maybe this rule needs tweaking…

  5. ^

     Even more controversial version of that take: maybe we should precommit to pin our regularization to the values of humanity as of 2026. If almost everyone in the world changes their values to something that 2026 humans actively hate, that likely means that society has been eaten by some crazy totalizing memeplex. On the other hand, if past civilizations had done this, they’d probably lock in values like worshipping God and wives being obedient to their husbands, so this idea clearly needs some work.

• Caleb Biddulph 21 Jul 2026 19:38 UTC

  20 points

  2

  on: CBid­dulph’s Shortform

  Interesting quote from Eliezer’s 1996 essay Staring into the Singularity, from the section about mind-uploading:

  Maybe, after Transcending, you will perhaps be a bit changed. If that is so, it is both inevitable and morally right. Given that absolutely nothing you can do will change that, why worry? Save your anxiety for what you can affect.

  (The same goes for worries about hostile superintelligences wiping out humanity. I cannot guarantee this won’t happen, since I’m not superintelligent. However, I am reasonably (95%) certain that the Powers will be ethical; I just don’t know what really is “ethical”, the true moral right. If wiping out your creators is morally wrong, it won’t happen. If we do get wiped out, it will be because, were we upgraded to superintelligences instead, we would see the moral necessity and commit suicide. If that’s the case, there is absolutely nothing we can do about it, both in the long run (see above) and the short run (see below); and, even were we granted the opportunity, it would be morally wrong to take it. Observer-dependent morality is a chimerical artifact of evolutionary competition; it dissolves under the application of sufficient intelligence. If you think it would be wrong to kill you, and you are rationally correct about it, presumably the Powers will see the same argument and you’ll be perfectly safe. Again, why worry?)

  I knew he used to believe this, since he mentioned it in My Childhood Death Spiral, but it’s funny seeing Eliezer Yudkowsky argue that he’s 95% certain ASI will be ethical. Not trying to dunk on Eliezer in particular; it makes me worry that anyone could make giant foundational errors that completely reverse the sign of their impact.

• Caleb Biddulph 6 Jul 2026 23:27 UTC

  3 points

  0

  in reply to: Linch’s comment on: Linch’s Shortform

  Guessing this is in reaction to @Liface’s shortform today?

• Caleb Biddulph 27 Jun 2026 23:09 UTC

  19 points

  −9

  in reply to: davekasten’s comment on: Ex­is­ten­tial AI safety needs an effec­tive so­cial move­ment. PauseAI is build­ing it

  Hm, I just spent a while reading Holly’s Twitter, and a lot of it appealed to me, especially the recent articles she’s been putting out. I like that her thinking is decorrelated from EA while still having a deep understanding of it.

  I don’t necessarily agree with her framing on everything, especially some things she says about people who genuinely care a lot about doing good. But IMO having someone like Holly around is probably net-positive. Trying on the frame “I and/​or my friends are complicit in evil by cooperating with/​working at AI labs” and noticing where/​why it feels uncomfortable or wrong seems like a useful exercise, even if I don’t actually end up believing it. It seems like Holly is playing the role of someone who confronts people with that frame. I don’t know if her particular style of Twitter-dunking is optimal for that goal, but it seems ~fine at least from what I’ve seen. (If she’s said anything especially line-crossing that I might want to know about, feel free to let me know.)

  Not sure whether I want to donate to PauseAI US in light of this. Maybe! I’ve been looking for things to donate to that I believe in, but which will not shortly be inundated with Anthropic money. Given Holly’s stance on working for Anthropic, PauseAI US may be an especially good fit for the latter criterion.