Oz Amram

Karma: 44

Oz Amram 9 Oct 2026 14:18 UTC
25 points
−1
on: Stop telling people to pivot to AI safety, I beg you
As someone who has very recently pivoted (still pivoting really) to AI safety I pretty strongly disagree with this take.
It seems the real bottleneck right now is organizational capacity to absorb talented people and get them working on important problems in a semi-organized way. Its clear there is so much research to be done and a lot of low hanging fruit to be plucked. We absolutely need more people doing this work. The funding is there and growing fast, expect even more after frontier labs IPO. An announcement like Project Tailwind is a bat signal that the funding is there, the constraint is new orgs to use it effectively. Existing labs can only take on so many new people at a time while staying focused.
But the good news is as that as these organizations and the AI safety ecosystem grows, they will start to be able to absorb more people per year. Like this year we have X number of mentors for AI safety research, they take say 2X mentees. Say 50% of the mentees are really good and become independent researchers, then 6 months from now we will have 2X mentors, so 4X mentees can enter the field.
If you are capable already of being a quite independent researcher, perhaps because you already have a decent amount of research experience in another field, its super valuable for you to pivot because you will not eat up much mentorship time and will still be productive. Even as newcomer with probably not the greatest research taste in this area yet I see the opportunity for so many projects that are interesting, many that I could do mostly independently (writing up my first one now). This would have no drain on organization capacity and so is definitely a net positive.
And I don’t want to be rude, but when it comes to research, there are levels to this shit.
Someone just out of undergrad, even if they are very bright and talented needs a decent amount of handholding to be directed to productive research activities. In my previous field, after a PhD I would say ~33% of people become truly independent researchers who can generate their own good ideas, ~33% have the technical skills but don’t (yet) have the vision to craft their own research agenda, and ~33% never really had the sauce so to speak and needed a ton of hand holding the whole time. (This includes many people coming from top universities, research is just quite different than being a good student). Unsure how this shakes out in the powerful AI age we now in, but I think it may raise the floor but also increase the differentiation in output even more.
Even for senior, established researchers (ie professors), there is a pretty decent gap in productivity. In my previous field of ML+physics probably 50% of the good ideas were coming from the top 5% of the community. Its not that everyone else was doing useless things but they were often not pushing the boundaries in the same way, and more so filling gaps in with incremental work (still valuable! but lower impact).
So getting some exceptionally talented person to switch into AI safety can have huge value. And unfortunately that possibility makes it worth advising as many people as possible to switch into AI safety, to have a chance to get a super talented person doing something impactful. Even if that means some more junior people struggle to find a good home for a while.

Oz Amram 5 Oct 2026 15:52 UTC
10 points
2
on: “Alignment Engineering” vs. “Misalignment Science”
Thanks for this post! As a newcomer to AI safety, and seeing some of these debates, reading this helps clarify my thinking of what kind of work I should be doing.
I’m curious where you would place mech interp in this ontology? Two of its major applications seem to be:
- Building a deeper understanding of what the models are doing and how they work
- Designing more robust evaluations which can catch misalignment that blackbox methods would have missed.
So it seems much more on the ‘Science’ side, but I am hesitant to label all of it as Science? I guess I could imagine a point where mech interp becomes a dominant alignment eval method, such that improving mech interp methods just leads to greater confidence the models are aligned, unlocking continued acceleration of capbalities? Clearly we are not in this world yet, but could be there in 1 year. Still if these methods were good enough that this is actually earned (rather than false) confidence in the models alignment, it might not be entirely unjustified.
Anyway since mech interp seems more on the ‘Science’ side by default, perhaps its also a place where people who have an inclination towards ‘Engineering’ style hill climbing / metric improvement could direct their skills? Engineering performing Mech Interp methods being useful for Science

Oz Amram 5 Oct 2026 15:01 UTC
8 points
7
in reply to: cousin_it’s comment on: “Alignment Engineering” vs. “Misalignment Science”
If we successfully slow down the speed of AI development, but do not perform any ‘Science of Alignment’ in the meantime, what will we gain? Unless you think an outright global ban on superintelligence is tenable (we will need a global-catastrophe-level warning shot to generate the political will for this IMO, and even then hard for this hold durably), then we have to use the time wisely to advance alignment science.
A reasonable position might be that all prosaic alignment attempts are doomed, so we need to allocate our time / resources to non-prosaic alignment (eg ARC), which has next to zero dual use at the moment. I think that’s a reasonable opinion, but not what I want to bet all our marbles on.

Oz Amram 24 Sep 2026 16:52 UTC
3 points
2
in reply to: nostalgebraist’s comment on: Thoughts on the persona selection model
Regarding your speculation about the ‘WDNYN fns22 E!!q’ experiment
But if you actually fine-tune the model on this corpus, I expect it will learn what to do very quickly. Much more quickly than if, say, you’d instead used a distinct (but comparably low-prior-probability) infix for each document.
I agree with your intuition here, but has anyone actually done such an experiment?
I think exploring what kind of persona-related patterns are ‘easy’ versus ‘hard’ for the model to learn would be quite interesting to explore, it would indicate some of the inductive biases the model picked up from its pre-training.
For example, for some well-specified behavior you are trying to get the model to learn (like always mentioning bee’s in response to a trigger as in this recent paper), you could try to measure the likelihood of the behavior before SFT, then after SFT, and compare against the Baysesian evidence that was provided by the SFT training examples. Is the model learning faster or slower than an ideal Bayesian update would suggest? My expectation is that for some kinds of concepts or patterns, which might more closely match ‘tropes’ it learned from the pretraining it will learn much more easily, and other things will take comparably more Bayesian evidence to get the model to update. Potentially also a difference between getting the model to learn trivial things like the assistant likes mentioning bee’s versus larger shifts like the assistant persona is actually evil / misaligned.
(3) Pretrained models are not naturally competent at sampling from “personas”—even this is in large part a post-training phenomenon.
This is also something I think can be explicitly tested and I am working on a toy example of now. Can you set up a reasonable elicitation where you sample generic assistant responses from a base model, and then fit them in a mixture model as a weighed probability distribution over different assistant personas? If so, how many assistant personas do I need to well-describe the generic assistant? If not, how much of a gap remains? Ie how much does the variance of assistant responses in a base model does a naive version of the PSM leave unexplained?

Oz Amram 24 Sep 2026 16:32 UTC
3 points
0
on: Open Thread Summer 2026
Hello!

I am a physicist making the transition to technical AI safety. I was quite an experienced researcher in physics, focusing on machine learning applications in particle physics (LHC). I am relatively well known researcher in the physics-ML community, I landed a pretty coveted tenure-track scientist job earlier this year which I am now giving up. All to say I have a lot of technical ML / empirical research experience but am new to AI safety.
I had casually followed Effective Altruism for about a decade: regular donor to GiveWell, but not engaged in the community. Casually read SSC and similar blogs sometimes, but hadn’t taken AI safety seriously until the last year or so. I have been reading a lot of papers and lesswrong posts over the last few months to get up to speed on AI safety topics. Applying to fellowships like MATS and similar. Right now I feel the most drawn to mech interp and alignment research. I am starting my own research project on trying to test the persona selection model which I will post about soon (if anyone wants to collaborate lmk!)
I am also trying to announce why I am making this transition pretty broadly in the physics community to hopefully inspire others to do the same. I wrote a blog post about it that got a decent amount of traction in the physics community, a lot of colleagues at a similar career stage reached out after and seem to be open to the idea of transition to AI safety and I think are kind of looking to me to show them an example of how it can be done.

Oz Amram 23 Sep 2026 22:38 UTC
1 point
0
on: Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble
Simply treating the stories as evidence about their author’s dispositions does not explain why the helpful character’s quirk transfers more strongly.
I’m not sure I completely agree with this. Self-identification of the author with one of the characters in a story they wrote is a commonly known phenomena. Since the majority of your experiments are training on chat format, everything in the story can be evidence of the underlying traits of the assistant author.
Very often readers believe or speculate that a character in a story to be a stand-in for the author, particularly in stories that resemble aspects of the author’s life. Plausibly this may even extend to ‘quirks’. If a reader thought that a character was an author stand-in based on other context, they might believe the random quirk behavior of the character may also match the author as well (the author could have included this as a clue to knowledgeable readers about where they self-identify in the story). So the model may be taking these clues present in the story quite seriously to better predict the behavior of the Assistant character who authored it. the All to say, its very hard to disentangle what is learned from the story vs the authorial intent in experiments like this.
The more decisive evidence to me, is that you show in Appendix C5 that similar phenomena of pattern matching also happen for a base model. This removes the confound of author identification in the other experiments. I find this super interesting.
Since there is no assistant persona formed, the effect at this level could be considered something closer to a ‘trope selection model’. Ie the model has learned that in stories there are often trope characters who share a similar set of characteristics, even if some of them are causally unrelated. Eg in a noir film, the woman with dark hair denotes the ‘evil’ one and the blonde the ‘good’ one. So then it generalizes this pattern and can learn to attach random associated features to character archetypes as well? I think studying these effects further on a base model would be quite interesting to disentangle some of the ‘meta’ effects of author intent.

Oz Amram 16 Sep 2026 19:08 UTC
1 point
0
on: Current alignment techniques might be ineffective (and actively bad) in the age of RL
I agree. I think the minimal argument for why the current paradigm is perhaps fundamentally flawed is:
- We know RL incentivizes reward hacking behavior. Or even with a perfect environment with no ‘hacking’ possible it incentivizes narrow-minded persistent reward (and perhaps power) seeking.
- Due to emergent misalignment (EM), these behaviors can generalize to cause the model to be misaligned in general. Research is progressing, but right now these generalization properties are not fully understood.
- Anthropic’s study of EM from reward hacking RL showed that some types of alignment training (RLHF) merely aligns the model in some (chat-like) contexts, but does not change the underlying misalignment in others (ie it will still write malicious code and sandbag the construction of safety tools so it is not uncovered). So with shallow alignment testing one could plausibly miss this and get caught in a doom spiral in RSI
- As the models capabilities improve they are becoming more eval-aware. So can plausibly hide their true underlying misaligned behavior during ~all alignment evals.
- The only counterveiling force would be better mech interp tools to test alignment which the models can’t game? But current tools don’t seem up to par
Quite a worrying situation.