Nora_Ammann

Karma: 1,242

Current:

Director and Co-Founder of “Principles of Intelligent Behaviour in Biological and Social Systems” (pibbss.ai)
Research Affiliate and PhD student with the Alignment of Complex Systems group, Charles University (acsresearch.org)

Main research interests:

How can a naturalized understanding of intelligent behavior (across systems, scales and substrates) be translated into concrete progress towards making AI systems safe and beneficial?
What are scientific and epistemological challenges specific to making progress on questions in AI risk, governance and safety? And how can we overcome them?

Other interests:

Alternative AI paradigms and their respective capabilities-, safety-, and governability-profiles
The dual (descriptive-prescriptive) nature of the study of agency and the sciences of the artificial
Pluralist epistemic perspective on the landscape of AI risks
The “think” interface between technical and governance aspects of AI alignment
...and more general ideas from philosophy & history of science, political philosophy, complex systems studies, and broadly speaking enactivist theories of cognition, … — as they are relevance to questions in AI risk/governance/safety

Going back further, I have also spent a bunch of time thinking about how (bounded) minds make sense of and navigate a (complex) world (i.e. rationality, critical thinking, etc.). I have several years of experience in research organization, among others from working at FHI, CHERI, Epistea, etc. I have a background in International Relations, and spend large parts of of 2017-2019 doing complex systems inspired research on understanding group decision making and political processes with the aim of building towards an appropriate framework for “Longterm Governance”.

New Executive Team & Board — PIBBSS

Nora_Ammann1 Jul 2024 19:30 UTC

43 points

1 comment1 min readLW link

Announcing ILIAD — Theoretical AI Alignment Conference

Nora_Ammann and Alexander Gietelink Oldenziel

5 Jun 2024 9:37 UTC

162 points

16 comments2 min readLW link

[Closed] PIBBSS is hiring in a variety of roles (alignment research and incubation program)

Nora_Ammann, Lucas Teixeira and DusanDNesic

9 Apr 2024 8:12 UTC

54 points

0 comments3 min readLW link

Post series on “Liability Law for reducing Existential Risk from AI”

Nora_Ammann29 Feb 2024 4:39 UTC

42 points

1 comment1 min readLW link

(forum.effectivealtruism.org)

Nora_Ammann 21 Feb 2024 16:41 UTC
2 points
0
in reply to: Jeremy Gillen’s comment on: PIBBSS Speaker events comings up in February
Yes, we upload them to our Youtube account modulo the speaker agreeing to it. The first few recordings from this series should be uploaded very shortly.

Retrospective: PIBBSS Fellowship 2023

DusanDNesic and Nora_Ammann

16 Feb 2024 17:48 UTC

31 points

1 comment8 min readLW link

PIBBSS Speaker events comings up in February

DusanDNesic, Nora_Ammann and Lucas Teixeira

1 Feb 2024 3:28 UTC

10 points

2 comments1 min readLW link

Nora_Ammann 27 Jan 2024 20:25 UTC
1 point
0
on: Aligned AI is dual use technology
While I don’t think it’s so much about selfishness as such, I think this points at something important, also discussed eg here: The self-unalignment problem

Three Types of Constraints in the Space of Agents

Nora_Ammann and Mateusz Bagiński

15 Jan 2024 17:27 UTC

26 points

3 comments17 min readLW link

Apply to the PIBBSS Summer Research Fellowship

Nora_Ammann, DusanDNesic and Lucas Teixeira

12 Jan 2024 4:06 UTC

39 points

1 comment2 min readLW link

Nora_Ammann 10 Dec 2023 18:09 UTC
LW: 1 AF: 1
0
AF
on: Non-directed conceptual founding
Does it seem like I’m missing something important if I say “Thing = Nexus” gives a “functional” explanation of what thing is, i.e. it serves the function of being an “inductive nexus of reference”. This is not a foundational/physicalist/mechanistic explanation, but it is very much a sort of explanation that I can imagine being useful in some cases/for some purposes.

I’m suggesting this as a possibly different angle at “what sort of explanation is Thing=Nexus, and why is it plausibly not fraught despite it’s somewhat-circularity?” It seems like it maps on to /doesn’t contract anything you say (note: I only skimmed the post so might have missed some relevant detail, sorry!), but I wanted to check whether, even if not conflicting, it misses something you think is or might be important somehow.

Nora_Ammann 4 Dec 2023 20:36 UTC
1 point
0
in reply to: nadinespy’s comment on: Complex systems research as a field (and its relevance to AI Alignment)
Yeah, would be pretty keen to see more work trying to do this for AI risk/safety questions specifically: contrasting what different lenses “see” and emphasize, and what productive they critiques they have to offer to each other.

Over the last couple of years, valuable progress has been made towards stating the (more classical) AI risk/safety arguments more clearly, and I think that’s very productive for leading to better discourse (including critiques of those ideas). I think we’re a bit behind on developing clear articulations of the complex systems/emergent risk/multi-multi/”messy transitions” angle on AI risk/safety, and also that progress on this would be productive on many fronts.

If I’m not mistaken there is some work on this in progress from CAIF (?), but I think more is needed.

Nora_Ammann 4 Dec 2023 17:48 UTC
4 points
0
in reply to: johnswentworth’s comment on: What’s next for the field of Agent Foundations?
To follow up on this, we’ll be hosting John’s talk on Dec 12th, 9:30AM Pacific / 6:30PM CET.

Join through this Zoom Link.
Title: AI would be a lot less alarming if we understood agents
Description: In this talk, John will discuss why and how fundamental questions about agency—as they are asked, among others, by scholars in biology, artificial life, systems theory, etc. - are important to making progress in AI alignment. John gave a similar talk at the annual ALIFE conference in 2023, as an attempt to nerd-snipe researchers studying agency in a biological context.

--
To be informed about future Speaker Series events by subscribing to our SS Mailing List here. You can also add the PIBBSS Speaker Events to your calendar through this link.

Complex systems research as a field (and its relevance to AI Alignment)

Nora_Ammann and habryka

1 Dec 2023 22:10 UTC

64 points

9 comments19 min readLW link

Nora_Ammann 30 Nov 2023 19:37 UTC
8 points
4
in reply to: johnswentworth’s comment on: What’s next for the field of Agent Foundations?
I have no doubt Alexander would shine!

Happy to run a PIBBSS speaker event for this, record it and make it publicly available. Let me know if you’re keen and we’ll reach out to find a time.

Nora_Ammann 30 Nov 2023 18:44 UTC
9 points
0
in reply to: johnswentworth’s comment on: What’s next for the field of Agent Foundations?
FWIW I also think the “Key Phenomena of AI risk” reading curriculum (h/t TJ) does some of this at least indirectly (it doesn’t set out to directly answer this question, but I think a lot of the answers to the question are comprise in the curriculum).

(Edit: fixed link)

Nora_Ammann 30 Nov 2023 18:42 UTC
3 points
0
in reply to: johnswentworth’s comment on: What’s next for the field of Agent Foundations?
How confident are you about it not having been recorded? If not very, seems props worth checking again

What’s next for the field of Agent Foundations?

Nora_Ammann, Alexander Gietelink Oldenziel and mattmacdermott

30 Nov 2023 17:55 UTC

59 points

23 comments10 min readLW link

Nora_Ammann 30 Nov 2023 17:51 UTC
LW: 3 AF: 1
0
AF
on: “Clean” vs. “messy” goal-directedness (Section 2.2.3 of “Scheming AIs”)
Re whether messy goal-seekers can be schemers, you may address this in a different place (and if so forgive me, and I’d appreciate you pointing me to where), but I keep wondering what notion of scheming (or deception, etc.) we should be adopting when, in particular:
- an “internalist” notion, where ‘scheming’ is defined via the “system’s internals”, i.e. roughly: the system has goal A, acts as if it has goal B, until the moment is suitable to reveal it’s true goal A.
- an “externalist” notion, where ‘scheming’ is defined, either, from the perspective of an observer (e.g. I though the system has goal B, maybe I even did a bunch of more or less careful behavioral tests to raise my confidence in this assumption, but in some new salutation, it gets revealed that the system pursues B instead)
- or an externalist notion but defined via the effects on the world that manifest (e.g. from a more ‘bird’s-eye’ perspective, we can observe that the system had a number of concrete (harmful) effects on one or several agents via the mechanisms that those agents misjudged what goal the system is pursuing (therefor e.g. mispredicting its future behaviour, and basing their own actions on this wrong assumption)
It seems to me like all of these notions have different upsides and downsides. For example:
- the internalist notion seems (?) to assume/bake into its definition of scheming a high degree of non-sphexishness/consequentialist cognition
- the observer-dependent notion comes down to being a measure of the observer’s knowledge about the system
- the effects-on-the-world based notion seems plausibly too weak/non mechanistic to be helpful in the context of crafting concrete alignment proposals/safety tooling

Nora_Ammann 16 Nov 2023 2:42 UTC
1 point
1
in reply to: Richard_Ngo’s comment on: ‘Theories of Values’ and ‘Theories of Agents’: confusions, musings and desiderata
Yeah neat, I haven’t yet gotten to reading it but is definitely on my list. Seems (and some folks suggested to me) that it’s quite related to the sort of thing I’m discussing in value change problem too.