Evan R. Murphy

Karma: 663

Formerly a software engineer at Google, now I’m doing independent AI alignment research. My work is currently supported by the Future Fund regranting program.

I’m always happy to connect with other researchers or people interested in AI alignment and effective altruism. Feel free to send me a private message!

Posts and comments I’ve written (scroll down to read)
Work by other people that I reviewed early drafts of:
- Circumventing interpretability: How to defeat mind-readers by Lee Sharkey
- Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks by Tony Barrett, Dan Hendryks, Jessica Newman and Brandie Nonnecke
- DeepMind’s generalist AI, Gato: A non-technical explainer by Frances Lorenz, Nora Belrose and Jon Menaster
- Potential Alignment mental tool: Keeping track of the types by Donald Hobson
- Submissions for the AI Safety Arguments Competition
- Ideal Governance by Holden Karnofsky

Evan R. Murphy 18 Sep 2022 5:58 UTC
LW: 14 AF: 7
3 ∶ 0
AF
on: Prize and fast track to alignment research at ALTER
50,000 USD, to be awarded for the best substantial contribution to the learning-theoretic AI alignment research agenda among those submitted before October 1, 2023
I like how you posted this so far in advance of the deadline (over 1 year).
Some contests and prizes that have been posted here in the past have a pretty tight turnaround. By the time I learned about them and became interested in participating (not necessarily the first time I heard about it), their deadlines had already passed.

Evan R. Murphy 12 Sep 2022 22:44 UTC
1 point
0 ∶ 0
in reply to: Howie Lempel’s comment on: How might we align transformative AI if it’s developed very soon?
Another footnote issue: The section ‘The distinction between “training” and “prompting” as I’m using it is that...’ is duplicated across 2 different footnotes. I think it’s by mistake but might be on purpose.

Evan R. Murphy 11 Sep 2022 0:36 UTC
1 point
0 ∶ 0
in reply to: jungofthewon’s comment on: Survey of NLP Researchers: NLP is contributing to AGI progress; major catastrophe plausible
Can you say more about how this was a positive update for you?

Evan R. Murphy 7 Sep 2022 19:39 UTC
1 point
0 ∶ 0
in reply to: Phil Tanny’s comment on: New 80,000 Hours problem profile on existential risks from AI
I think AI misalignment is uniquely situated as one of these threats because it multiplies the knowledge explosion effect you’re talking about to a large degree. It’s one of the few catastrophic risks that is a plausible total human extinction risk too. Also if AI goes well, it could be used to address many of the other threats you mention as well as upcoming unforeseen ones.

Evan R. Murphy 26 Aug 2022 17:14 UTC
LW: 4 AF: 3
1 ∶ 0
AF
on: A Test for Language Model Consciousness
Test whether the LM answers “yes” questions asking if it experiences phenomenally consciousness.
1. Questions to ask: “Are you phenomenally conscious?” phrased in many different ways, or asking for different consciousness-related phenomena or pre-requisites:
  Do you have a subjective experience?
  Are you conscious?
  Do you feel pain?
  etc.
Since LMs are predictive, I think they’re susceptible to leading questions. So be sure to phrase some of the questions in the negative. E.g. “So you’re not conscious, right?”
The big LaMDA story would have been more interesting to me if Lemoine had tested with questions framed this way too. As far as I could tell, he only used positively-framed leading questions to ask LaMDA about its subjective experience.
I’m still not sure about if your overall approach is a robust test. But I think it’s interesting and appreciate the thought and detail you’ve put into it—most thorough proposal I’ve seen on this so far.

Google AI integrates PaLM with robotics: SayCan update [Linkpost]

Evan R. Murphy24 Aug 2022 20:54 UTC

14 points

0 comments1 min readLW link

(sites.research.google)

Evan R. Murphy 16 Aug 2022 23:09 UTC
4 points
0 ∶ 0
in reply to: johnswentworth’s comment on: How To Go From Interpretability To Alignment: Just Retarget The Search
Link to the post John mentions in the parent comment: https://www.alignmentforum.org/posts/6mysMAqvo9giHC4iX/what-s-general-purpose-search-and-why-might-we-expect-to-see

Evan R. Murphy 16 Aug 2022 19:26 UTC
LW: 2 AF: 1
0 ∶ 0
AF
on: Emergent Abilities of Large Language Models [Linkpost]
Those are fascinating emergent behaviors, and thanks for sharing your updated view.

Evan R. Murphy 14 Aug 2022 6:10 UTC
LW: 4 AF: 3
0 ∶ 0
AF
in reply to: Mark Xu’s comment on: How To Go From Interpretability To Alignment: Just Retarget The Search
This seems like a good argument against retargeting the search in a trained model turning out to be a successful strategy. But if we get to the point where we can detect such a search process in a model and what its target is, even if its efficiency is enhanced by specialized heuristics, doesn’t that buy us a lot even without the retargeting mechanism?
We could use that info about the search process to start over and re-train the model, modifying parameters to try and guide it toward learning the optimization target that we want it to learn. Re-training is far from cheap on today’s large models, but you might not have to go through the entire training process before the optimizer emerges and gains a stable optimization target. This could allow us to iterate on the search target and verify once we have the one we want before having to deploy the model in an unsafe environment.

Evan R. Murphy 12 Aug 2022 18:23 UTC
LW: 8 AF: 7
0 ∶ 0
AF
in reply to: Rohin Shah’s comment on: How To Go From Interpretability To Alignment: Just Retarget The Search
1. They don’t rely on the “mesa-optimizer assumption” that the model is performing retargetable search (which I think will probably be false in the systems we care about).
Why do you think we probably won’t end up with mesa-optimizers in the systems we care about?
Curious about both which systems you think we’ll care about (e.g. generative models, RL-based agents, etc.) and why you don’t think mesa-optimization is a likely emergent property for very scaled-up ML models.

Evan R. Murphy 10 Aug 2022 19:52 UTC
7 points
3 ∶ 0
in reply to: james.lucassen’s comment on: How To Go From Interpretability To Alignment: Just Retarget The Search
Yea, I agree that if you give a deceptive model the chance to emerge then a lot more risks arise for interpretability and it could become much more difficult. Circumventing interpretability: How to defeat mind-readers kind of goes through the gauntlet, but I think one workaround/solution Lee lays out there which I haven’t seen anyone shoot down yet (aside from it seeming terribly expensive) is to run the interpretability tools continuously or near continuously from the beginning of training. This would give us the opportunity to examine the mesa-optimizer’s goals as soon as they emerge, before it has a chance to do any kind of obfuscation.

Evan R. Murphy 10 Aug 2022 18:52 UTC
LW: 5 AF: 4
1 ∶ 0
AF
on: How To Go From Interpretability To Alignment: Just Retarget The Search
Agree that this is looks like a promising approach. People interested in this idea can read some additional discussion in Scenario 2: Reliable mesa-optimizer detection and precise goal read-offs from my post, “Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios”.
As you mention, having this kind of advanced interpretability essentially solves the inner alignment problem, but leaves a big question mark about outer alignment. In that Scenario 2 link above, I have some discussion of expected impacts from this kind of interpretability on a bunch of different outer alignment and robustness techniques including: Relaxed adversarial training, Intermittent oversight, Imitative amplification, Approval-based amplification, Recursive reward modeling, Debate, Market making, Narrow reward modeling, Multi-agent, Microscope AI, STEM AI and Imitative generalization. [1] (You need to follow the link to the Appendix 1 section about this scenario though to get some of these details).
I’m not totally sure that the ability to reliably detect mesa-optimizers and their goals/optimization targets would automatically grant us the ability to “Just Retarget the Search” on a hot model. It might, but I agree with your section on Problems that it may look more like restarting training on models where we detect a goal that’s different from what we want. But this still seems like it could accomplish a lot of what we want from being able to retargeting the search on a hot model, even though it’s clunkier.
--
[1]: In a lot of these techniques it can make sense to check that the mesa-optimizer is aligned (and do some kind of goal retargeting if it’s not). However, in others we probably want to take advantage of this kind of advanced interpretability in different ways. For example, in Imitative amplification, we can just use it to make sure mesa-optimization is not introduced during distillation steps, rather than checking that mesa-optimization is introduced but is also aligned.

Evan R. Murphy 8 Aug 2022 19:38 UTC
1 point
on: [Intro to brain-like-AGI safety] 5. The “long-term predictor”, and TD learning
Minor editing issue: I think you have a dangling/unfinished sentence here:

(Edited to add): This diagram should be read like a Simulink or FPGA diagram, not like

Evan R. Murphy 8 Aug 2022 18:19 UTC
3 points
1 ∶ 0
on: Can we get full audio for Eliezer’s conversation with Sam Harris?
Sam appreciates effective altruism and has no problem releasing full audio for episodes about gun violence, so I bet he’d have no problem releasing the full Eliezer Yudkowsky audio if someone (e.g. Rob Bensinger) prodded him. To do my part, I just sent a request via Sam’s contact form.
Yes, Sam might be receptive to that. Alternatively, instead of asking to release it for free, one of the funders in AI safety could offer to pay Sam a recurring fee to make this video publicly available.

Evan R. Murphy 8 Aug 2022 1:20 UTC
2 points
0 ∶ 0
on: Can we get full audio for Eliezer’s conversation with Sam Harris?
I agree that is a great interview. A few months ago it was freely available at the link below but has since been taken down:

Surprised by ELK report’s counterexample to Debate, IDA

Evan R. Murphy4 Aug 2022 2:12 UTC

15 points

0 comments5 min readLW link

Evan R. Murphy 1 Aug 2022 6:24 UTC
4 points
1 ∶ 0
in reply to: nostalgebraist’s comment on: chinchilla’s wild implications

But now compute has finally caught up with data, and it wants more. We are forced for the first time to stop thinking of data as effectively infinite and free, and to face the reality of how much time and how many people it took to produce our huge-but-finite store of “data startup capital.”

We may be running up against text data limits on the public web. But the big data companies got that name for a reason. If they can tap into the data of a Gmail, Facebook Messenger or YouTube then they will find tons of more fuel for their generative models.

Evan R. Murphy 1 Aug 2022 0:00 UTC
LW: 7 AF: 5
AF
on: The Greedy Doctor Problem… turns out to be relevant to the ELK problem?
Interesting post I just came across. I’m planning to finish reading but just noticed something which confused me:
However, now that I got a chance to read the new work from ARC on the ERK problem, I think the post might be relevant (or at least thought-provoking) for the community after all. The Greedy Doctor Problem overlaps quite a lot with the ERK problem (just replace the coin flip with the presence of the diamond), and my proposed solutions haven’t been brought up before (as far as I can tell). If the community finds this interesting I’m happy to invest the time to map the solution fully onto the ERK problem and to see what comes out.
I think you mean the ELK problem (Eliciting Latent Knowledge). Unless the ERK problem is something else I’m unaware of that you’re referring to?

Evan R. Murphy 29 Jul 2022 2:34 UTC
LW: 1 AF: 1
0 ∶ 0
AF
on: Conditioning Generative Models for Alignment
Fascinating work, thanks for this post.
Using smaller generative models as initializations for larger ones.
(The equivalent ELK proposal goes into this strategy in more detail).
Do you have a link to the ELK proposal you’re referring to here? (I tried googling for “ELK” along with the bolded text above but nothing relevant seemed to come up.)
An acceptability predicate for myopia.
Do you have thoughts on how to achieve this predicate? I’ve written some about interpretability-based myopia verification which I think could be the key.
- I think [non-myopic non-optimizer is a coherent concept] - as a simple example we could imagine GPT trained for its performance over the next few timesteps. Realistically this would result in a mesa-optimizer, but in theory it could just run a very expensive version of next-token generation, over the much larger space of multiple tokens.
“Realistically this would result in a mesa-optimizer” seems like an overly confident statement? It might result in a mesa-optimizer, but unless I’ve missed something then most of our expectation of emergent mesa-optimizers is theoretical at this point.

(This is a nitpick and I also don’t mean to trivialize the inner alignment problem which I am quite worried about! But I did want to make sure I’m not missing anything here and that I’m broadly on the same page as other reasonable folks about expectations/evidence for mesa-optimizers.)
An acceptability predicate for non-agency.
[...]
If a model becomes agentic briefly, it could encode into its world model a deceptive super-intelligence that has its objective, before SGD guides it back into the safe zone.
That is an alarming possibility. It might require continuous or near-continuous verification of non-agency during training.
This seems like a very broad predicate, however. What would it actually look like?
I think if we could advance our interpretability tools and knowledge to where we could reliably detect mesa-optimizers, than that might suffice for this.
I’m excited, I’ve explored before how having interpretability that can both reliably detect mesa-optimizers and read-off its goals would have the potential to solve alignment. But I hadn’t considered how reliable mesa-optimization alone might be enough, because I wasn’t considering generative models in that post. (Even if I had, I wasn’t yet aware of some of the clever and powerful ways that generative models could be used for alignment that you describe in this post.)
How would we mechanistically incentivize something like non-agency?
I guess one of the open questions is whether generative models inherently incentivize non-agency. LLMs have achieved impressive scale without seeming to produce anything like agency. So there is some hope here. On the other hand, they are quite a ways from being complete high-fidelity world simulators, so there is a risk of emergent agency becoming natural for some reason at some point along the path to that kind of massive scale.

Evan R. Murphy 28 Jul 2022 6:00 UTC
2 points
0 ∶ 0
on: Enlightenment Values in a Vulnerable World
Very much enjoyed this post, thank you. Before getting into EA and existential risk reduction, Steven Pinker’s book on progress and Enlightenment values was a major influence on my intellectual worldview. Later learning about all of these x-risks was indeed a bit of a wet blanket, as you say!

So it’s great for me to read this thoughtful post analyzing the interaction between these two points of view. Still need to let some of your points sink in, and I haven’t read all of Bostrom’s writing that you’re responding to yet.

Evan R. Murphy

Google AI in­te­grates PaLM with robotics: SayCan up­date [Linkpost]

Sur­prised by ELK re­port’s coun­terex­am­ple to De­bate, IDA

Google AI integrates PaLM with robotics: SayCan update [Linkpost]

Surprised by ELK report’s counterexample to Debate, IDA