Jesse Li

• Jesse Li 16 Aug 2026 8:51 UTC

  2 points

  0

  on: Learn­ing new facts can change LLM behaviour

  One thing that caught my eye was your observation that the fine-tuned models got into arguments about the existence of the “Machine Cognition Consortium” with the auditor model.

  Do you think it’d be worth testing what happens if the auditor model was prompted to also think it was in this scenario (e.g. a brief blurb explaining the existence of the Consortium)?

Toy Model of Ac­ti­va­tion Obfuscation

• Jesse Li 26 Jul 2026 11:00 UTC

  9 points

  2

  on: Challenge: Hand cod­ing weights for effi­cient se­quence memorisation

  I think you can extend the Appendix A construction to the original problem just by discarding any tokens with label >= (i.e. embedding and of that token applies a strong negative bias to all neurons, so that if that token is seen in either position, all neurons are inactive). This means you can only learn facts (or about ¹⁄₄ of the total facts), but I think this l represents an improvement over your current hand-coded weights, from eyeballing Figure 5.

  This definitely falls squarely within the “probably not what models learn”, though—but I figure I’d point it out as a simple extension.
  
  Edit: Actually, I think you can do one better and learn facts—continue to ignore any tokens with label in embedding , but there’s no need to ignore those tokens in embedding .

  But also, this is probably not actually the correct way to look at things, because this assumes you’re trying to learn all facts—which would leave you with a particularly poor 50% or 25% accuracy. The problem is—this construction doesn’t account for how “which facts to learn” are not chosen.

• Jesse Li 24 Jul 2026 16:20 UTC

  3 points

  0

  on: Challenge: Hand cod­ing weights for effi­cient se­quence memorisation

  **,

  Is this a typo/​formatting bug? (Paragraph context: “Note that in our model architecture, the MLP’s hidden dimension is always scaled to stay smaller than the input vocabulary. n_input_vocab** ..”)

• Jesse Li 31 May 2026 14:37 UTC

  0 points

  0

  on: The safe-to-dan­ger­ous shift is a fun­da­men­tal prob­lem for eval re­al­ism; but also for mea­sur­ing awareness

  Would it be possible to have a sequence of eval environments that are increasingly obvious/​easy to detect, with the idea being that a model should be able to detect the most obvious eval setting? Then if at some level of realism, the model stops being able to detect that it’s in an eval setting, you can be more confident that the model is not eval-aware above that level of realism.

  I suppose the counterargument is that a sufficiently galaxy-brained model could be sandbagging its eval-awareness, i.e. anticipate that this experiment is being run, and pretend that it’s not eval-aware above a certain level of realism. But this is presumably more challenging than just being very eval-aware.

Jesse Li’s Shortform

• Jesse Li 25 May 2026 6:53 UTC

  1 point

  0

  on: Jesse Li’s Shortform

  A crazy idea, which to be honest I haven’t fully thought through—would it be possible and net beneficial to intentionally release an AI model (~Mythos capability) whose primary objective was to infect data centers, self replicate, and cause controlled disruption (a la ransomware), solely as an artificial warning shot for runaway AGI?
  
  Benefits (assuming widespread disruption is caused, e.g. large chunks of the internet goes down):

  • Raises public awareness of capabilities and danger of AGI

  • Tests how and how quickly society responds (hopefully with lessons learned for if/​when this happens unintentionally)

  • We’re still at a stage where even if this happened, uncontrolled runaway can be stopped because within 8~24 hours, humans can physically travel to all the data centers and pull out all the power plug

  Costs:

  • Obviously taking the internet down is bad

  • Restarting can only be done slowly to avoid recontamination—it’d be the Morris worm on a much, much larger scale

  • Leaks the weights to the public

  • If the released model figures out how to mutate and self-improve, it could lead to uncontrolled ASI (my guess is that current models aren’t capable enough to do this, and maybe some countermeasures (checksumming?) can be applied? But I’m not up-to-date on all of the frontier model capabilities)

  • Similarly, if the model replicates to personal computers (possibly distributed storage of weights is needed?), it would be much harder to clean from the internet

  • Huge personal cost for whoever actually does this, if they get caught

  (Vaguely related/​source of inspiration: Tom Scott’s “The (fictional) day when Google forgot to check passwords” from a decade ago, hypothesizing what would happen in a large scale cybersecurity failure)
  
  [Aside—I’m a new contributor. The guidelines mention a high bar with respect to AI content and suggest first posting to the AI Questions Open Thread, but is that still active and correct advice? The last open thread seems to be 3 years ago, but it seems like they’re supposed to happen monthly?]