Curt Tigges

• Curt Tigges 6 Apr 2026 1:42 UTC

  3 points

  10

  on: Steer­ing Might Stop Work­ing Soon

  I think this hypothesis might benefit from more rigorous evaluation. To be clear, I think it’s entirely possible that you could be right, but we need more evidence that more sophisticated methods won’t work. Your tests were with the simplest version of steering, something akin to 14th-century surgery. I think we need much more rigorous evidence from mech interp.
  
  For example, perhaps the issue is merely that larger models have more complex representation manifolds, and thus simple additive steering pushes activations off-manifold. Or plausibly we need better steering vectors—I think that it’s likely we’ll have much more sophisticated things to steer with than sparse dictionary learning features (and indeed many such things are already in the literature).
  
  In other words, I think you need to show some reason why our steering methods’ sophistication can’t evolve with model scale and complexity.

• Curt Tigges 15 Apr 2025 21:47 UTC

  7 points

  0

  on: To be leg­ible, ev­i­dence of mis­al­ign­ment prob­a­bly has to be behavioral

  I’m not sure I entirely agree with the overall recommendation for researchers working on internals-based techniques. I do agree that findings will need to be behavioral initially in order to be legible and something that decision-makers find worth acting on.

  My expectation is that internals-based techniques (including mech interp) and techniques that detect specific highly legible behaviors will ultimately converge. That is:

  1. Internals/​mech interp researchers will, as they have been so far at least in model organisms, find examples of concerning cognition that will be largely ignored or not acted on fully

  2. Eventually, legible examples of misbehavior will be found, resulting in action or increased scrutiny

  3. This scrutiny will then propagate backwards to finding causes or indicators of that misbehavior, and provided interp tools are indeed predictive, this path that has been developed in parallel will suddenly be much more worth paying attention to

  Thus, I think it’s worth progressing on these internals-based techniques even if their use isn’t immediately apparent. When legible misbehaviors arrive, I expect internals-based detection or analysis to be more directly applicable.

• Curt Tigges 14 Jan 2025 22:51 UTC

  4 points

  0

  on: How do you deal w/​ Su­per Stim­uli?

  I use Freedom and Limit on my computer and Stay Focused on my Android phone. The former two allow for a combination of complete blocking during certain time windows and time limits (for any website, even across browsers and even if you open an incognito window). The latter does both for my phone.
  
  I block all social media and content during prime working hours and implement a 30-minute limit outside of that. It works pretty well. I may make it more strict because I sometimes find myself looking at Twitter, etc. occasionally when watching a TV show in the evenings.
  
  I also use BlockTube to get rid of YouTube Shorts entirely from my web browser. They no longer show up in search results or in the menu.
  
  Finally, I recommend the tools here, though I haven’t tried all of them: https://​​liamrosen.com/​​2023/​​04/​​18/​​modding-social-media-to-win-the-attention-war/​​

• Curt Tigges 12 Dec 2024 19:28 UTC

  6 points

  1

  on: Some­thing Is Lost When AI Makes Art

  I find this argument quite compelling, and this is also why I find the idea of “AI girl/​boyfriends” largely uninteresting. Without actual connection to another mind (that has experiences and phenomenal consciousness), any of these things—art, deep conversations about thoughts/​feelings, what have you—eventually falls flat. (That includes one-way connection through art).

SAEBench: A Com­pre­hen­sive Bench­mark for Sparse Autoencoders

• Curt Tigges 10 Dec 2024 18:28 UTC

  4 points

  4

  on: Zen and The Art of Semi­con­duc­tor Man­u­fac­tur­ing

  I quite enjoyed reading this. Very evocative.
  
  Welcome to San Francisco.

Stitch­ing SAEs of differ­ent sizes

• Curt Tigges 18 Jun 2024 18:49 UTC

  1 point

  0

  on: The Best Tacit Knowl­edge Videos on Every Subject

  Domain: Software engineering, mech interp
  
  Bryce Meyer (primary maintainer of TransformerLens, and software engineer with many years of experience) has a weekly coding stream event where he does live coding on TransformerLens—resolving bugs, adding features and tests, etc. I’ve found it to be useful!
  
  You can find it in the Open Source Mechanistic Interpretability Slack, under the “code-sessions” channel (feel free to DM for an invite).

• Curt Tigges 30 May 2024 22:02 UTC

  9 points

  9

  on: Ta­lent Needs of Tech­ni­cal AI Safety Teams

  Great post, but there is one part I’d like to push back on:

  Iterators are also easier to identify, both by their resumes and demonstrated skills. If you compare two CVs of postdocs that have spent the same amount of time in academia, and one of them has substantially more papers (or GitHub commits) to their name than the other (controlling for quality), you’ve found the better Iterator. Similarly, if you compare two CodeSignal tests with the same score but different completion times, the one completed more quickly belongs to the stronger Iterator.

  This seems like a bit of an over-claim. I would endorse a weaker claim, like “in the presence of a high volume of applicants, CodeSignal tests, GitHub commits, and paper count statistically provide some signal,” but the reality of work in the fields of research and software development is often such that there isn’t a clean correspondence between these measures and someone’s performance. In addition, all three of these measures are quite easy to game (or Goodhart).

  For example, in research alone, not every paper entails the same-sized project; two high-quality papers could have an order of magnitude difference in the amount of work required to produce them. Not every research bet pays off, too—some projects don’t result in papers, and research management often plays a role in what directions get pursued (and dropped or not if they are unproductive). There are also many researchers who have made a career out of getting their names on as many papers as possible; there is an entire science to doing this that is completely independent of your actual research abilities.

  In the case of CodeSignal evaluations, signal is likewise relatively low-dimensional and primarily conveys one thing: enough experience with a relatively small set of patterns that one can do the assessment very quickly. I’ve taken enough of these and seen enough reviews from senior engineers on CodeSignal tests to know that they capture only a small, specific part of what it takes to be a good engineer, and overemphasize speed (which is not the main thing you want from an actual senior engineer; you want quality as well as maintainability and readability, which often are at odds with speed. Senior engineers’ first instinct is not generally to jump in and start spitting out lines of code like their lives depend on it). Then there’s the issue of how hackable/​gameable the assessments are; senior engineer Yanir Seroussi has a good blog post on CodeSignal specifically: https://​​yanirseroussi.com/​​2023/​​05/​​26/​​how-hackable-are-automated-coding-assessments/​​

  I’m definitely not arguing that these metrics are useless, however. They do provide some signal (especially if the volume of applicants is high), but I’d suggest that we see them as imperfect proxies that we’re forced to use due to insufficient manpower for comprehensive candidate evaluations, rather than actually capturing some kind of ground truth.

• Curt Tigges 15 May 2024 18:51 UTC

  1 point

  0

  on: my note system

  Perhaps more important than these details: How do you curate input to take notes on, and what is the purpose you take the notes for? How do you use the notes once written? (This latter point seems to be one of the biggest reason many people have dropped PKM systems.)

• Curt Tigges 11 May 2024 5:35 UTC

  1 point

  0

  in reply to: Vanessa Kosoy’s comment on: Dat­ing Roundup #3: Third Time’s the Charm

  Very kind of you to say. :) I think for me, though, the source of the emotion I felt when reading this series was something like: “Ah, so in addition to ensuring we are dateable ourselves, we must fix society, capitalism (at least the dating part of it), culture, etc. in order to have a Good Dating Universe.” Which in retrospect was a bit overblown of me, so I think I no longer endorse the strong version of what I said in that comment.

• Curt Tigges 10 May 2024 5:50 UTC

  6 points

  2

  on: Dat­ing Roundup #3: Third Time’s the Charm

  I think this list may successfully convince some to stay off the dating market indefinitely. Who in the world has time to work on all of this? At best, this is just a massive set of to-dos; at worst, it’s an enormous list of all the ways the dating world sucks and reasons why you’ll fail.
  
  Upon reflection: This is a good collection of information, even if it is rather discouraging to read. May we all find exceptions to the unfortunate trends that seem to characterize the modern dating landscape.

• Curt Tigges 5 Feb 2024 19:47 UTC

  4 points

  2

  on: Why I no longer iden­tify as transhumanist

  I actually went through the same process as what you describe here, but it didn’t remove my “transhumanist” label. I was a big fan of Humanity+, excited about human upgrading, etc. etc. I then became disillusioned about progress in the relevant fields, started to understand nonduality and the lack of a persistent or independent self, and realized AI was the only critical thing that actually was in the process of happening.
  
  In that sense, my process was similar but I still consider myself a transhumanist. Why? Because for me, solving death or trying to make progress in the scientific fields that lead to various types of augmentations aren’t the biggest or most critical pieces of transhumanism. One could probably have been a transhumanist in the 1800s, because for me it’s about the process of imagining and defining and philosophizing about what humanity—on an individual organism level as well as on a sociocultural level—will become (or what it might be worthwhile to become) after particular types of technological transitions.

  Admittedly, there is a normative component that’s something like “those of us who want to should be able to become something more than base human” and isn’t really active until those capabilities actually exist, but the process of thinking about what it might be worthwhile to become, or what the transition will be like, or what matters and what is valuable in this kind of future, are all important.

  It’s not about maximizing the self, either—I’m not an extropian. Whether or not something called “me” exists in this future (which might be soon but might not), the conscious experience of beings in it matter to me (and in this sense I’m a longtermist).

  Will an aligned AI solve death? Maybe, but my hopes don’t rely on this. Humanity will almost certainly change in diverse ways, and is already changing a bit (though often not in great ways). It’s worthwhile to think about what kind of changes we would want to create, given greater powers to do so.

• Curt Tigges 7 Apr 2023 16:35 UTC

  1 point

  0

  in reply to: Thomas Kwa’s comment on: Ex­plo­ra­tory Anal­y­sis of RLHF Trans­form­ers with TransformerLens

  Yes, tuned lens is an excellent tool and generally superior to the original logit lens. In this particular case, I don’t think it would show very different results, however (and in any case the logit lens is only a small part of the analysis), but I think it would be interesting to have some kind of integration with TransformerLens that enabled the training and usage of tuned lens as well.

Ex­plo­ra­tory Anal­y­sis of RLHF Trans­form­ers with TransformerLens

• Curt Tigges 8 Jan 2023 23:31 UTC

  11 points

  7

  on: I tried to learn as much Deep Learn­ing math as I could in 24 hours

  This is a cool idea, and I have no doubt it helped somewhat, but IMO it falls prey to the same mistake I see made by the makers of almost every video series/​online course/​list of resources for ML math: assuming that math is mostly about concepts and facts.

  It’s only about 5% that. Maybe less. I and many others in ML have seen the same videos and remembered the concepts for a while too. And forgotten them, in time. More than once! On the other hand, I’ve seen how persistently and operationally fluent (especially in ML and interpretability) people become when they actually learned math the way it must be learned: via hundreds of hours of laborious exercises, proofs, derivations, etc. Videos and lectures are a small fraction of what’s ultimately needed.

  For most of ML, it’s probably fine—you’ll never need to do a proof or do more than simple linear algebra operations by hand. But if you want to do the really hard stuff, especially in interpretability, I don’t think there’s any substitute for cranking through those hours.

  To be clear, I think this weekend was a great start on that—if you continue immediately to taking full courses and doing the exercises. I’m a top-down learner, so it would certainly help me. But unless it’s practiced in very short order, it will be forgotten, and just become a collection of terms you recognize when others talk about them.