CallumMcDougall

ARENA 8.0 - Call for Applicants

An­nounc­ing Gemma Scope 2

Trans­mit­ting Misal­ign­ment with Sublimi­nal Learn­ing via Paraphrasing

How Can In­ter­pretabil­ity Re­searchers Help AGI Go Well?

A Prag­matic Vi­sion for Interpretability

• CallumMcDougall 3 Oct 2025 7:36 UTC

  25 points

  10

  on: Ome­las Is Perfectly Misread

  Try to find any article, paper, or blog post that lays out this reading. I haven’t found one (but there is sometimes someone in the comments who seems to get it).

  Omelas: How We Talk About Utopia

  Pretty much identical thesis, does this count?

ARENA 7.0 - Call for Applicants

• CallumMcDougall 9 Sep 2025 22:47 UTC

  27 points

  15

  in reply to: davekasten’s comment on: chanamess­inger’s Shortform

  True, although I wish more people would engage with the common anti-AI-x-risk argument of “tech CEOs are exaggerating existential risk because they think it’ll make their products seem more important and potentially world changing, and so artificially boost hype”. Not saying I agree with this, but there’s at least some extent to which it’s true, and I think this community often fails to appropriately engage with and combat this argument.

  In general, this is why “appeal to authority” arguments should generally be avoided if we’re talking about people who are widely seen as untrustworthy and having ulterior motives. At most I think people like Geoffrey Hinton are seen as reputable and not as morally compromised so serve as better subjects for an appeal to authority, but mostly rather than needing to appeal to authority at all we should just try and bring things back to the object-level arguments.

ARENA 6.0 - Call for Applicants

New Cause Area Proposal

Nega­tive Re­sults for SAEs On Down­stream Tasks and Depri­ori­tis­ing SAE Re­search (GDM Mech In­terp Team Progress Up­date #2)

• CallumMcDougall 22 Feb 2025 15:59 UTC

  3 points

  0

  in reply to: yihe’s comment on: In­duc­tion heads—illustrated

  Sorry I didn’t get to this message earlier, glad you liked the post though! The answer is that attention heads can have multiple different functions—the simplest way is to store things entirely orthogonally so they lie in fully independent subspsaces, but even this isn’t necessary because it seems like transformers take advantage of superposition to represent multiple concepts at once, more so than they have dimensions.

ARENA 5.0 - Call for Applicants

Scal­ing Sparse Fea­ture Cir­cuit Find­ing to Gemma 9B

• CallumMcDougall 1 Jan 2025 18:34 UTC

  5 points

  0

  in reply to: Fabien Roger’s comment on: How to repli­cate and ex­tend our al­ign­ment fak­ing demo

  Oh, interesting, wasn’t aware of this bug. I guess this is probably fine since most people replicating it will be pulling it rather than copying and pasting it into their IDE. Also this comment thread is now here for anyone who might also get confused. Thanks for clarifying!

• CallumMcDougall 30 Dec 2024 14:40 UTC

  3 points

  0

  on: How to repli­cate and ex­tend our al­ign­ment fak­ing demo

  +1, thanks for sharing! I think there’s a formatting error in the notebook, where the tags like <OUTPUT> were all removed and replaced with empty strings (e.g. see attached photo). We’ve recently made the ARENA evals material public, and we’ve got a working replication there which I think has the tags in the right place (section 2 of 3 on the page linked here)

  

SAEBench: A Com­pre­hen­sive Bench­mark for Sparse Autoencoders

• CallumMcDougall 6 Dec 2024 10:28 UTC

  6 points

  1

  on: [Paper] A is for Ab­sorp­tion: Study­ing Fea­ture Split­ting and Ab­sorp­tion in Sparse Autoencoders

  Amazing post! Forgot to do this for a while, but here’s a linked diagram explaining how I think about feature absorption, hopefully ppl find it helpful!

  

• CallumMcDougall 12 Oct 2024 21:05 UTC

  4 points

  0

  in reply to: chanind’s comment on: Toy Models of Fea­ture Ab­sorp­tion in SAEs

  I don’t know of specific examples, but this is the image I have in my head when thinking about why untied weights are more free than tied weights:

  

  I think more generally this is why I think studying SAEs in the TMS setup can be a bit challenging, because there’s often too much symmetry and not enough complexity for untied weights to be useful, meaning just forcing your weights to be tied can fix a lot of problems! (We include it in ARENA mostly for illustration of key concepts, not because it gets you many super informative results). But I’m keen for more work like this trying to understand feature absorption better in more tractible cases

AI Align­ment Re­search Eng­ineer Ac­cel­er­a­tor (ARENA): Call for ap­pli­cants v4.0