I work on applied mathematics and AI at the Johns Hopkins University Applied Physics Laboratory. I also do AI safety related work for the Johns Hopkins Institute for Assured Autonomy. I am currently doing a CS PhD focused on AI safety at the University of Maryland, Baltimore County.
Aryeh Englander
Karma: 250
[Question] Is MIRI’s reading list up to date?
[External Event] 2022 IEEE Conference on Assured Autonomy (ICAA)
Yes, this sounds like a reasonable interpretation.
I am reluctant to mention specific examples, partly because maybe I’ve misunderstood and partly because I hate being at all confrontational. But regardless, I have definitely seen this outside the rationalist community, and I have definitely noticed myself doing this. Usually I only do it in my head though, where I feel upset when it’s coming from outside my group but if someone inside the group says it then I’ll mentally nod along.
Failure mode I think I’ve noticed, including among rationalists (and certainly myself!): If someone in your in-group criticizes something about the group, then people often consider that critique to be reasonable. If someone outside the group levels the exact same criticism, then that feels like an attack on the group, and your tribal defensiveness kicks into gear, potentially making you more susceptible to confirmation / disconfirmation bias or the like. I’ve noticed myself and I’m pretty sure others in the rationalist community doing this, and even reacting in clearly different ways to the exact same critique when we hear it from an in-group member or someone outside the group.
Do you think this is correct or off the mark? Also, is there a name for this and have their been studies about it?
[Question] Improving rationality / scout mindset on a debate group
NIST AI Risk Management Framework request for information (RFI)
Modelling Transformative AI Risks (MTAIR) Project: Introduction
[Question] How would the Scaling Hypothesis change things?
I’d like to hear more thoughts, from Rohin or anybody else, about how the scaling hypothesis might affect safety work.
[Question] Pros and cons of working on near-term technical AI safety and assurance
New post on the EA Forum: Some AI Governance Research Ideas
Just came across this: Research ideas to study humans with AI Safety in mind
[Question] List of good AI safety project ideas?
Thanks Adam for setting this up! I have no idea if my experience is representative, but that was definitely one of the highest-quality discussion sessions I’ve had at events of this type.
I don’t think this is quite an example of a treacherous turn, but this still looks relevant:
Lewis et al., Deal or no deal? end-to-end learning for negotiation dialogues (2017):
Analysing the performance of our agents, we find evidence of sophisticated negotiation strategies. For example, we find instances of the model feigning interest in a valueless issue, so that it can later ‘compromise’ by conceding it. Deceit is a complex skill that requires hypothesising the other agent’s beliefs, and is learnt relatively late in child development (Talwar and Lee, 2002). Our agents have learnt to deceive without any explicit human design, simply by trying to achieve their goals.
(I found this reference cited in Kenton et al., Alignment of Language Agents (2021).)
[Link] Whittlestone et al., The Societal Implications of Deep Reinforcement Learning
Institute for Assured Autonomy (IAA) newsletter
That’s later in the linked wiki page: https://timelines.issarice.com/wiki/Timeline_of_AI_safety#Full_timeline
Thanks!