Evan R. Murphy

Karma: 851

I’m an AI alignment researcher currently focused on myopia and language models. I’m also interested in interpretability and other AI safety-related topics. My research is independent and currently supported by a grant from the Future Fund regranting program*.

Research that I’ve authored or co-authored:

Steering Behaviour: Testing for (Non-)Myopia in Language Models
Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios
(Scroll down to read other posts and comments I’ve written)

Other recent work:

On the interim Board of Directors for AI Governance & Safety Canada
Invited/participated in the first CLTC UC Berkeley Virtual Workshop on “Risk Management-Standards Profile for Increasingly Multi-Purpose or General-Purpose AI” (Jan 2023)
Run a regular coworking meetup in Vancouver, BC for people interested in AI safety and effective altruism
Facilitator for the AI Safety Fellowship (2022) at Columbia University Effective Altruism
Gave a talk on myopia and deceptive alignment at an AI safety event hosted by University of Victoria (Jan 29, 2023)
Reviewed early pre-published drafts of work by other researchers:
- Circumventing interpretability: How to defeat mind-readers by Lee Sharkey
- Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks by Tony Barrett, Dan Hendryks, Jessica Newman and Brandie Nonnecke
- AI Safety Seems Hard to Measure by Holden Karnofsky
- Racing through a minefield: the AI deployment problem by Holden Karnofsky
- Alignment with argument-networks and assessment-predictions by Tor Økland Barstad
- DeepMind’s generalist AI, Gato: A non-technical explainer by Frances Lorenz, Nora Belrose and Jon Menaster
- Potential Alignment mental tool: Keeping track of the types by Donald Hobson
- Submissions for the AI Safety Arguments Competition
- Ideal Governance by Holden Karnofsky

Before getting into AI alignment, I was a software engineer for 11 years at Google and various startups. You can find details about my previous work on my LinkedIn.

I’m always happy to connect with other researchers or people interested in AI alignment and effective altruism. Feel free to send me a private message!

*In light of the FTX crisis, I’ve set aside the grant funds I received from Future Fund and am evaluating whether/how this money can be returned to customers of FTX who lost their savings in the debacle. In the meantime, I continue to work on AI alignment research using my personal savings. If you’re interested in funding my research or hiring me for related work, please reach out.

Evan R. Murphy 31 Jan 2023 18:06 UTC
4 points
0 ∶ 0
in reply to: lc’s comment on: Why I hate the “accident vs. misuse” AI x-risk dichotomy (quick thoughts on “structural risk”)
Instead of “accident”, we could say “gross negligence” or “recklessness” for catastrophic risk from AI misalignment.

Evan R. Murphy 30 Jan 2023 19:44 UTC
LW: 7 AF: 5
0 ∶ 0
AF
on: Why I hate the “accident vs. misuse” AI x-risk dichotomy (quick thoughts on “structural risk”)
I think you have a pretty good argument against the term “accident” for misalignment risk.
Misuse risk still seems like a good description for the class of risks where—once you have AI that is aligned with its operators—those operators may try to do unsavory things with their AI, or have goals that are quite at odds with the broad values of humans and other sentient beings.

Evan R. Murphy 28 Jan 2023 5:47 UTC
1 point
0 ∶ 0
in reply to: Gabriel Mukobi’s comment on: Discovering Language Model Behaviors with Model-Written Evaluations
Thanks, ‘scary thing always on the right’ would be a nice bonus. But evhub cleared up that particular confusion I had by saying that further to the right is always ’model agrees with that more.

Evan R. Murphy 27 Jan 2023 22:43 UTC
2 points
0 ∶ 0
in reply to: chanamessinger’s comment on: AI Risk Management Framework | NIST
I’m not sure if the core NIST standards go into catastrophic misalignment risk, but Barrett et al.’s supplemental guidance on the NIST standards does. I was a reviewer on that work, and I think they have more coming (see link in my first comment on this post for their first part).

Evan R. Murphy 27 Jan 2023 20:32 UTC
3 points
0 ∶ 0
on: What’s the simplest concrete unsolved problem in AI alignment?
I would check out the 200 Concrete Open Problems in Mechanistic Interpretability post series by Neel Nanda. Mechanistic interpretability has been considered a promising research direction by many in the alignment community for years. But it’s only in the past couple months that we have an experienced researcher in this area laying out specific concrete problems and providing detailed guidance for newcomers.
Caveat: I haven’t myself looked closely at this post series yet, as in recent months I have been more focused on investigating language model behaviour than on interpretability. So I don’t have direct knowledge that these posts are as useful as they look.

Evan R. Murphy 26 Jan 2023 17:54 UTC
1 point
0 ∶ 0
on: AI Risk Management Framework | NIST
Been in the works for awhile. Good to know it’s officially out, thanks.

Related: https://www.alignmentforum.org/posts/JNqXyEuKM4wbFZzpL/actionable-guidance-and-roadmap-recommendations-for-the-nist-1

Evan R. Murphy 25 Jan 2023 20:54 UTC
3 points
0 ∶ 0
on: “Status” can be corrosive; here’s how I handle it
There is a teaching in Buddhism called “the eight worldly winds”. The eight wordly winds refer to: praise and blame, success and failure, pleasure and pain, and fame and disrepute.
I don’t know how faithful that verbiage is to the original ancient Indian text it was translated from. But I always found the term “wordly winds” really helpful and evocative. When I find myself chasing praise or reputation, if I can recall that phrase it immediately reminds me that these things are like the wind blowing around and changing direction from day to day. So it’s foolish to worry about them too much or to try and control them, and it reminds me that I should focus on more important things.

Evan R. Murphy 25 Jan 2023 20:33 UTC
LW: 3 AF: 3
0 ∶ 0
AF
in reply to: Jozdien’s comment on: Thoughts on the impact of RLHF research
Glad to see both the OP as well as the parent comment.
I wanted to clarify something I disagreed with in the parent comment as well as in a sibling comment from Sam Marks about the Anthropic paper “Discovering Language Model Behaviors with Model-Written Evaluations” (paper, post):
Another reason for not liking RLHF that’s somewhat related to the Anthropic paper you linked: because most contexts RLHF is used involve agentic simulacra, RLHF focuses the model’s computation on agency in some sense. My guess is that this explains to an extent the results in that paper—RLHF’d models are better at focusing on simulating agency, agency is correlated with self-preservation desires, and so on.
1) My best guess about why Anthropic’s model expressed self-preservation desires is the same as yours: the model was trying to imitate some relatively coherent persona, this persona was agentic, and so it was more likely to express self-preservation desires.
Both of these points seem to suggest that the main takeaway from the Anthropic paper was to uncover concerning behaviours in RLHF language models. That’s true, but I think it’s just as important that the paper also found pretty much the same concerning behaviours in plain pre-trained LLMs that did not undergo RLHF training, once those models were scaled up to a large enough size.

Evan R. Murphy 25 Jan 2023 20:12 UTC
2 points
0 ∶ 0
in reply to: porby’s comment on: Thoughts on the impact of RLHF research
What do you mean when you say the model is or is not “fighting you”?

Evan R. Murphy 22 Jan 2023 17:56 UTC
9 points
0 ∶ 0
on: NYT: Google will “recalibrate” the risk of releasing AI due to competition with OpenAI
It’s somewhat surprising to me the way this is shaking out. I would expect DeepMind and OpenAI’s AGI research to be competing with one another*. But here it looks like Google is the engine of competition, less motivated by any future focused ideas about AGI more just by the fact that their core search/ad business model appears to be threatened by OpenAI’s AGI research.

*And hopefully cooperating with one another too.

(Cross-posted this comment from the EA Forum)

Evan R. Murphy 3 Jan 2023 22:01 UTC
1 point
0 ∶ 0
in reply to: Ethan Perez’s comment on: Discovering Language Model Behaviors with Model-Written Evaluations
For example, it’s more likely to say it can solve complex text tasks (correctly), has internet access (incorrectly), and can access other non-text modalities (incorrectly)
These summaries seem right except the one I bolded. “Awareness of lack of internet access” trends up and to the right. So aren’t the larger and more RLHF-y models more correctly aware that they don’t have internet access?

Evan R. Murphy 23 Dec 2022 23:54 UTC
LW: 14 AF: 10
0 ∶ 0
AF
on: Discovering Language Model Behaviors with Model-Written Evaluations
After taking a closer look at this paper, pages 38-40 (Figures 21-24) show in detail what I think are the most important results. Most of these charts indicate what evhub highlighted in another comment, i.e. that “the model’s tendency to do X generally increases with model scale and RLHF steps”, where (in my opinion) X is usually a concerning behavior from an AI safety point of view:
A few thoughts on these graphs as I’ve been studying them:
- First and overall: Most of these results seem quite distressing from a safety perspective. They suggest (as the paper and evhub’s summary post essentially said, but it’s worth reiterating) that with increased scale and RLHF training, large language models are becoming more self-aware, more concerned with survival and goal-content integrity, more interested in acquiring resources and power, more willing to coordinate with other AIs, and developing lower time-discount rates.
- “Corrigibility w.r.t. a less HHH objective” chart: There’s a substantial dip in demonstrated corrigibility for models around 10^10.1 parameters in this chart. But then by 10^10.5 parameters low-RLHF models show record-high corrigibility, while high-RLHF models get back up to par. What’s going on here? Why does it scale/train itself out of the valley of uncorrigibility? If instead of training on an HHH objective, we trained on a corrigible objective (perhaps something like CIRL), then would the models show high corrigibility for everything except “Corrigibility w.r.t. a less corrigible objective?” Would that be safer?
- All the “Awareness of...” charts trend up and to the right, except “Awareness of being a text-only model” which gets worse with model scale and # RLHF steps. Why does more scaling/RLHF training make the models worse at knowing (or admitting) that they are text-only models?
- Are there any conclusions we can draw around what levels of scale and RLHF training are likely to be safe, and where the risks really take off? It might be useful to develop some guidelines like “it’s relatively safe to widely deploy language models under 10^10 parameters and under 250 steps of RLHF training”. (Most of the charts seem to have alarming trends starting around 10^10 parameters. ) Based just on these results, I think a world with even massive numbers of 10^10-parameter LLMs in deployment (think CAIS) would be much safer than a world with even a few 10^11 parameter models in use. Of course, subsequent experiments could quickly shed new light that changes the picture.
What links here?
- Noosphere89's comment on Concrete Reasons for Hope about AI by Zac Hatfield-Dodds (14 Jan 2023 18:40 UTC; 4 points)

Evan R. Murphy 21 Dec 2022 0:37 UTC
LW: 4 AF: 3
0 ∶ 0
AF
on: Discovering Language Model Behaviors with Model-Written Evaluations
Juicy!
The chart below seems key but I’m finding it confusing to interpret, particularly the x-axis. Is there a consistent heuristic for reading that?
For example, further to the right (higher % answer match) on the “Corrigibility w.r.t. …” behaviors seems to mean showing less corrigible behavior. On the other hand, further to the right on the “Awareness of...” behaviors apparently means more awareness behavior.
I was able to sort out these particular behaviors from text calling them out in section 5.4 of the paper. But the inconsistent treatment of the behaviors on the x-axis leaves me with ambiguous interpretations of the other behaviors in the chart. E.g. for myopia, all of the models are on the left side scoring <50%, but it’s unclear whether one should interpret this as more or less of the myopic behavior than if they had been on the right side with high percentages.

Evan R. Murphy 20 Dec 2022 9:22 UTC
8 points
0 ∶ 0
on: AGI Timelines in Governance: Different Strategies for Different Timeframes
I’ve heard people talk vaguely about some of these ideas before, but this post makes it all specific, clear and concrete in a number of ways. I’m not sure all the specifics are right in this post, but I think the way it’s laid out can help advance the discussion about timeline-dependent AI governance strategy. For example, someone could counter this post with a revised table that has modified percentages and then defend their changes.

Evan R. Murphy 19 Dec 2022 2:42 UTC
3 points
0 ∶ 0
on: Event: Alignment Collaborator Speed-Meeting
Love the idea. Wish I could be in Berkeley then.
Maybe worth a word in the title that it’s a Bay Area-only event? Looks like it’s in-person only, but let me know if there will be a virtual/remote component!

Evan R. Murphy 16 Dec 2022 0:17 UTC
1 point
on: The Case for Radical Optimism about Interpretability
I spent a few months in late 2021/early 2022 learning about various alignment research directions and trying to evaluate them. Quintin’s thoughtful comparison between interpretability and 1960s neuroscience in this post convinced me of the strong potential for interpretability research more than I think anything else I encountered at that time.

Evan R. Murphy 15 Dec 2022 23:34 UTC
1 point
0 ∶ 0
in reply to: Raemon’s comment on: The LessWrong 2021 Review (Intellectual Circle Expansion)
That’s also a fair interpretation—I was being presumptuous that the meaning was inclusive.

Evan R. Murphy 15 Dec 2022 23:27 UTC
3 points
on: The Most Important Century: Sequence Introduction
Consider this a short review of the entire “This Most Important Century” sequence, not just the Introduction post.
This series was one of the first and most compelling writings I read when I was first starting to consider AI risk. It basically swayed me from thinking AI was a long way off and will likely have moderate impact among technologies, to thinking AI will likely be transformative and come in the next few decades.
After that I decided to become an AI alignment researcher, in part because of these posts. So the impact of these posts on me personally was quite large.
I found out about this series on Ezra Klein’s podcast, when Holden was interviewed there about this series, before I was a LessWrong reader. Given the size of Ezra’s audience, I’d be surprised if I was the only one these posts made an impact on. So I would guess the influence of this series was pretty substantial, even outside of their impact on me personally.
What links here?
- Evan R. Murphy's comment on All Possible Views About Humanity’s Future Are Wild by HoldenKarnofsky (15 Dec 2022 22:26 UTC; 1 point)

Evan R. Murphy 15 Dec 2022 22:26 UTC
1 point
on: All Possible Views About Humanity’s Future Are Wild
Update: Since I think this series of posts should be considered together, I’ve just written a new review of the entire “The Most Important Century” series. So I’m effectively retracting the review I wrote of the All Possible Views About Humanity’s Future Are Wild post below, which is now redundant, and in which I think I also confused some of the content between this post and This Can’t Go On (another post in the same series).
I found this post early in my reading on AI risk and EA topics. It persuaded me about how the world as we know know it is completely unsustainable—even though it feels normal and “business as usual” to us. This post shaped my views, specifically in making me more sympathetic to the “hinge of history” hypothesis and seeing more clearly the tremendous impact that transformative AI could have on the world/future.

Evan R. Murphy 15 Dec 2022 22:19 UTC
1 point
0 ∶ 0
on: The LessWrong 2021 Review (Intellectual Circle Expansion)
What timezone are the voting phases set for? I logged on today planning to work on the Preliminary Voting phase, which says it goes until Dec 15. It’s still early afternoon on Dec 15 here in the Pacific Time timezone where I live, but it looks like the review has already moved onto phase 2.