Yonatan Cale

• Yonatan Cale 18 Nov 2025 23:57 UTC

  4 points

  0

  on: An­thropic is (prob­a­bly) not meet­ing its RSP se­cu­rity commitments

  I don’t even know whether I think Anthropic having good security is good or bad for the world

  Could you share the TL;DR for why this might be bad for the world?

• Yonatan Cale 13 Sep 2025 17:32 UTC

  5 points

  0

  on: Agen­tic Misal­ign­ment: How LLMs Could be In­sider Threats

  I recently read through the entire pdf—big thumbs up from me!

  A few questions /​ requests:

  1. Would you release the transcripts of part/​all the runs? (I’m not sure if there’s a consideration against doing this that I’m missing)

     1. I’m most interested in “no goal” runs that still showed misaligned behavior

  2. Would you share recordings of the computer-use if you have them?

     1. And also the setup for reproducing the computer-use scenario? (I’m guessing you have some code for a fake browser which isn’t in your repo)

  3. FYI, I think most people don’t know that you ran a variation where the model wasn’t given any goal (which is different from “no goal conflict”, which implies the existing and successor models have the same goal)

  Big thumbs up again!

• Yonatan Cale 12 Sep 2025 3:50 UTC

  2 points

  0

  on: Yonatan Cale’s Shortform

  TL;DR: Can we collect real-world examples of misalignment while preserving privacy?

  Naively, startups that monitor agents for unintended behavior could publish examples of misalignment they found, but I doubt almost any org would agree to this, since examples might involve the org’s code or other private data.

  Ideas:

  1. Work with orgs/​agents that don’t handle private info. Maybe open source repos?

  2. Only share the scenarios with a few research centers (Redwood?)

     1. Would orgs agree^[1] to this?

  3. When misalignment is found, ask the org if it’s ok to share it. If even 1% agree it still might be nice.

  4. Look up what created the culture of disclosing vulnerabilities in cyber security. Maybe that can be reproduced? (I do think the situation is very different here)

  5. Sharing an anonymized/​redacted version of the scenario

     1. Seems a lot less useful to me (unless there is enough info to reproduce similar behavior), but it would be nice if we could get a lot of these

  The Replit agent that deleted a production DB

  I found this interesting because the user of Replit seems to want to disclose the problem.

  I can imagine having a norm (or a regulation?) where after Replit fix whatever problem they had—they release the LLM transcripts that are relevant to the incident.

  (or maybe I’m wrong and this person wouldn’t want to share enough details for a full transcript)

  1. ^

     Looking at Cursor’s Privacy Policy:

     We do not use Inputs or Suggestions to train our models unless: (1) they are flagged for security review (in which case we may analyze them to improve our ability to detect and enforce our Terms of Service)

     But who decides what is flagged? And can Cursor do things other than train their models in other cases?

     Also, this is from their TOS:

     5.4. Usage Data. Anysphere may: (i) collect, analyze, and otherwise process Usage Data internally for its business purposes, including for security and analytics, to enhance the Service, and for other development and corrective purposes

     I’m not sure if this is a reliable hint for what orgs would agree to

• Yonatan Cale 9 Sep 2025 3:46 UTC

  6 points

  1

  on: The LLM Has Left The Chat: Ev­i­dence of Bail Prefer­ences in Large Lan­guage Models

  This seems like the highest quality source of data for “what do LLMs want” I’ve heard of (am I missing others?), may I download and analyze it myself?

  Thanks for doing this!

• Yonatan Cale 24 Jul 2025 11:23 UTC

  2 points

  0

  in reply to: Kaj_Sotala’s comment on: On “ChatGPT Psy­chosis” and LLM Sycophancy

  Fair,

  Note they used GPT-3 which wasn’t trained with RLHF (right?)

• Yonatan Cale 3 Jul 2025 11:57 UTC

  3 points

  0

  on: Con­sider chilling out in 2028

  Hey, I think I’m the target audience of this post, but it really doesn’t resonate well with me.

  Here are some of my thoughts/​emotions about it:

  First, things I agree with:

  1. Feeling doom/​depressed/​sad isn’t a good idea. I wish I’d have less of this myself :( it isn’t helping anyone

  2. Many people thought the world would end and they were wrong—this should give us a lower prior on ourselves being right about the world maybe ending

  3. Biases are probably in play

  4. I read you as trying to be supportive, which I appreciate, thanks

  Things that feel bad about:

  5. Was worrying about covid also similarly misguided?

  At the beginning of covid, people said “stop making a big deal about it”, “lots of people freaked out about things in the past and were wrong”, I also live close to a city with a lot of (religious) anti-vax people, and I visited a (hippie) place where I told someone when I was last tested and what I did since, but she told me she sees I look good and I better not think about it, I’ll be healthier if I don’t think about covid too much.

  Should I have listened to the people who told me to chill about covid? I assume you agree I shouldn’t have, beyond (I agree) no reason to emotionally freak out, just take reasonable actions, that’s all.

  What’s the difference between the AI and covid scenarios? I assume you’d say “covid was real” (?). Anyway, that’s my crux. Is AI danger “real” (vaguely defined).

  What if the covid deniers would say “if you’re so sure about covid, tell us exactly in what week (or whatever) everyone will die”—this seems like a hard/​unfair question. I personally think people spend way too much energy geeking out about AI timelines, and after (metaphorically) a million requests for timelines, a serious team wrote ai-2027, which is a probability distribution, where they think there’s a 50% chance problems will happen earlier, and 50% they’ll happen later (right?). Telling me to chill at 2028 seems similar to asking for an estimate for when covid problems will actually hit, getting a median probability distribution, and telling me to “chill” if that median passed (while still things seem to be getting worse for similar reasons?). Yeah there’s a time where if AI doesn’t seem to be taking over then I’ll eat my hat and say I am completely missing something about how the world works, but that time is not 2028.

  6. Fallacy Fallacy

  I often think back on The Adventures Of Fallacy Man.

  Both “sides” have fallacies. Maybe doomers want a way to deal with our mortality. Maybe non-doomers have a bias against imagining changes or whatever. I don’t think this kind of conversation is a good way to figure out what is true. I think AI is a complicated topic that makes it clear how problematic many rules-of-thumb or metaphors are and encourages us to look at the details, or at least to pick our mental shortcuts wisely.

• Yonatan Cale 1 Jul 2025 15:45 UTC

  2 points

  0

  on: Yonatan Cale’s Shortform

  Would you leave more anonymous feedback for people who ask for it if there was a product that did things like:

  1. Rewrite your feedback to make it more anonymous (with an LLM?)

  2. Aggregate your feedback with other feedback this person received, and only tell them things like “many people said you’re rude”

  3. Delay your feedback by some (random?) amount of time to make it less recognizable

  I’m mostly interested to hear from people who consider leaving feedback and sometimes don’t, I think it would be cool if we could make progress on solving whatever painpoint you have (not sure we’ll succeed, but worth trying).

  If you prefer to reply anonymously (very meta!) you can say so here. I’ll assume it’s ok to share responses as comments to this shortform unless you ask me not to.

  Idea credit: cwbakerlee

• Yonatan Cale 8 Apr 2025 12:34 UTC

  2 points

  0

  in reply to: Yonatan Cale’s comment on: Yonatan Cale’s Shortform

  So FYI, the emoji I posted above was actually AI generated.

  The real one looks like this (on my browser) :

  

  I still think the original one looks a lot like a paperclip, but I cheated and exaggerated this in the April Fool’s version

• Yonatan Cale 1 Apr 2025 16:45 UTC

  63 points

  0

  on: Yonatan Cale’s Shortform

  Seems like Unicode officially added a “person being paperclipped” emoji:

  

  Here’s how it looks in your browser: 🙂‍↕️

  Whether they did this as a joke or to raise awareness of AI risk, I like it!

  Source: https://​​emojipedia.org/​​emoji-15.1

• Yonatan Cale 5 Mar 2025 18:54 UTC

  1 point

  0

  in reply to: lisathiergart’s comment on: Yonatan Cale’s Shortform

  Hey,

  On anti-trust laws, see this comment. I also hope to have more to share soon

• Yonatan Cale 3 Mar 2025 21:46 UTC

  1 point

  0

  in reply to: Yonatan Cale’s comment on: Yonatan Cale’s Shortform

  I asked Claude how relevant this is to protecting something like a H100, here are the parts that seem most relevant from my limited understanding:

  What the paper actually demonstrates:

  1.⁠ ⁠Reading (not modifying) data from antifuse memory in a Raspberry Pi RP2350 microcontroller
  2.⁠ ⁠Using Focused Ion Beam (FIB) and passive voltage contrast to extract information
  

  Key differences between this and modifying an H100 GPU:

  1. 3D Transistor Structures: Modern 5nm chips use FinFET or GAAFET 3D structures rather than planar transistors. The critical parts are buried within the structure, making them fundamentally more difficult to access without destroying them.

  2. Atomic-Scale Limitations: At 5nm, we’re approaching atomic limits (silicon atoms are ~0.2nm). The physics of matter at this scale creates fundamental boundaries that better equipment cannot overcome.

  3. Ion Beam Physics: Even with perfect equipment, ion beams create interaction volumes and damage zones that become proportionally larger compared to the target features at smaller nodes.

• Yonatan Cale 3 Mar 2025 9:40 UTC

  1 point

  0

  in reply to: Jonathan_H’s comment on: Yonatan Cale’s Shortform

  Thanks! Is this true for a somewhat-modern chip that has at least some slight attempt at defense, or more like the chip on a raspberry pi?

• Yonatan Cale 28 Feb 2025 5:08 UTC

  5 points

  3

  in reply to: Mikhail Samin’s comment on: Mikhail Samin’s Shortform

  (Could you link to the context?)

• Yonatan Cale 27 Feb 2025 22:17 UTC

  17 points

  0

  on: Yonatan Cale’s Shortform

  Patching security problems in big old organizations involves problems that go a lot beyond “looking at code and changing it”, especially if aiming for a “strong” solution like formal verification.

  TL;DR: Political problems, code that makes no sense, problems that would be easy to fix even with a simple LLM that isn’t specialized on improving security.

  The best public resource I know is about this is Recoding America.

  Some examples iirc:

  1. Not having a clear primary key to identify people with.

  2. Having a website (a form) that theoretically works but doesn’t run on any browser that people actually use.

  3. Having a security component which is supposed to catch fake submissions of forms, but is way more likely to catch real submissions (it is imo net negative).

  I also learned some surprising things from working on fixing/​rewriting a major bank in Israel. I can’t share such juicy stories as Recoding America publicly, but here are some that I can:

  1. “written in kobol” is maybe ~1% of the problem and imo not an interesting pain point to focus on

  2. Many systems are microservices (much harder to define the expected functionality of an async system)

     1. Different parts of the bank are written using different technologies

  3. An example problem you’d find in the code is “representing amounts of money using javascript floats”

     1. It’s not complicated to fix, technically

     2. But people might say “we’re used to using floats, why change it?”

     3. Making changes might mean they’ll have to run manual tests, so they’ll be against it

     4. Some teams just don’t like having others touch their code, and maybe some care about their job security

  4. Another example easy thing to fix is “use enums”

     1. I’ve seen more than one conversations where “agreeing to use an enum” was a big political debate

  5. Sometimes, to understand a system, I’d do a combination of “looking at the code”, “looking at the docs (word documents with the design)”, and “talking to one or more of the people in charge of the system” (sometimes different people have different parts of the picture)

  [written with the hope that orgs trying to patch security problems will do well]

• Yonatan Cale 20 Feb 2025 18:46 UTC

  1 point

  0

  in reply to: Milan W’s comment on: Yonatan Cale’s Shortform

  I want the tool to proactively suggest things while working on the document, optimizing for “low friction for getting lots of comments from the LLM”. The tool you suggested does optimize for this property very well

• Yonatan Cale 20 Feb 2025 2:16 UTC

  1 point

  0

  in reply to: Milan W’s comment on: Yonatan Cale’s Shortform

  1. This is very cool, thanks!

     1. I’m tempted to add Claude support

  2. It isn’t exactly what I’m going for. Example use cases I have in mind:

     1. “Here’s a list of projects I’m considering working on, and I’m adding curxes/​considerations for each”

     2. “Here’s my new alignment research agenda” (can an AI suggest places where this research is wrong? Seems like checking this would help the Control agenda?)

     3. “Here’s a cost-effectiveness analysis of an org”

• Yonatan Cale 20 Feb 2025 0:54 UTC

  11 points

  1

  on: Yonatan Cale’s Shortform

  Things I’d suggest to an AI lab CISO if we had 5 minutes to talk

  1 minute version:

  • I think there are projects that can prepare the lab for moving to an air gapped network (protecting more than model weights) which would be useful to start early, would have minimal impact on developer productivity, and could be (to some extent) delegated^[1]

  Extra 4 minutes:

  Example categories of such projects:

  1. Projects that take serial time but can be done without the final stage that actually hurts developer productivity

     1. Toy example: Add extra ethernet cables to the building but don’t use them yet

  2. Reduce uncertainty about the problems that will be caused by a future security measure

     1. Toy example: Prepare for a (partially?) air gapped network by monitoring (with consent^[2]) which domains employees use and finding alternatives to them, e.g:

        1. Wikipedia --> Download it

        2. Social media --> Buy some employees a personal use computer, see if they like it?

        3. … each domain becomes a project to prioritize and delegate, hopefully

  3. Projects that require “product market fit” with the engineers

     1. Toy example: The lab wants a secure^[3] way to access model weights^[4]. They can try an MVP solution (github PRs?), get user feedback (“too much friction!”), and work on the next draft while the users go back to accessing the weights however they want.

        1. Note how much more it would hurt productivity if we’d wait with this project until security became critical and we’d have to force the engineers to use whatever solution we could come up with quickly. This is a common property of many projects I’d suggest.

  1. ^

     I’m assuming the CISO’s team has limited focus, but spending this focus on delegating projects is a good deal. I’m also assuming this is a problem they’re happy to solve with money.

  2. ^

     I endorse communicating why you want to do this and getting employee agreement, not just randomly following them

  3. ^

     e.g monitored

  4. ^

     I’m aware this example is more focused on model weights, but it felt shorter to write than other product-market-fit examples. e.g I think “experiment with opening a new office for employees who like to WFH” is more realistic for an air gapped network but was longer for me to explain

• Yonatan Cale 17 Feb 2025 23:04 UTC

  8 points

  2

  on: Yonatan Cale’s Shortform

  I’m looking for an AI tool which feels like Google Docs but has an LLM proactively commenting/​suggesting things.

  (Is anyone else interested in something like this?)

• Yonatan Cale 7 Feb 2025 23:42 UTC

  3 points

  0

  on: Plan­ning for Ex­treme AI Risks

  This post helped me notice I have incoherent beliefs:

  1. “If MAGMA self-destructs, the other labs would look at it with confusion/​pity and keep going. That’s not a plan”

  2. “MAGMA should self-destruct now even if it’s not leading!”

  I think I’ve been avoiding thinking about this.

  So what do I actually expect?

  If OpenAI (currently in the lead) would say “our AI did something extremely dangerous, this isn’t something we know how to contain, we are shutting down and are calling other labs NOT to train over [amount of compute], and are not discussing the algorithm publicly because of fear the open source community will do this dangerous thing, and we need the government ASAP”, do I expect that to help?

  Maybe?

  Probably nation states will steal all the models+algorithm+slack as quickly as they can, probably a huge open source movement will protest, but it still sounds possible (15%?) that the major important actors would listen to this, especially if it was accompanies by demos or so?

  What if Anthropic or xAI or DeepSeek (not currently in the lead) would shut down now?

  ...I think they would be ignored.

  Does that imply I should help advance the capabilities of the lab most likely to act as you suggest?

  Does this imply I should become a major player myself, if I can? If so, should I write on my website that I’m open to a coordinated pause?

  Should I give up on being a CooperateBot, given the other players have made it so overwhelmingly clear they are happy to defect?

  This is painful to think about, and I’m not sure what’s the right thing to do here.

  Open to ideas from anyone.

  Anyway, great post, thanks

• Yonatan Cale 1 Feb 2025 19:18 UTC

  1 point

  0

  in reply to: Yonatan Cale’s comment on: Yonatan Cale’s Shortform

  OpenAI already have this in their charter:

  We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”