Nora_Ammann

• Nora_Ammann 9 May 2023 18:26 UTC

  6 points

  0

  in reply to: Leon Lang’s comment on: An­nounc­ing “Key Phenom­ena in AI Risk” (fa­cil­i­tated read­ing group)

  Indeed that wasn’t intended. Thanks a lot for spotting & sharing it! It’s fixed now.

• Nora_Ammann 9 May 2023 0:53 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: Chris_Leong’s comment on: An­nounc­ing “Key Phenom­ena in AI Risk” (fa­cil­i­tated read­ing group)

  Good point! We are planning to gauge time preferences among the participants and fix slots then. What is maybe most relevant, we are intending to accommodate all time zones. (We have been doing this with PIBBSS fellows as well, so I am pretty confident we will be able to find time slots that work pretty well across the globe.)

An­nounc­ing “Key Phenom­ena in AI Risk” (fa­cil­i­tated read­ing group)

The space of sys­tems and the space of maps

An­nounc­ing the 2023 PIBBSS Sum­mer Re­search Fellowship

Reflec­tions on the PIBBSS Fel­low­ship 2022

In­ter­nal com­mu­ni­ca­tion framework

Maps and Blueprint; the Two Sides of the Align­ment Equation

In­tel­li­gent be­havi­our across sys­tems, scales and substrates

PIBBSS (AI al­ign­ment) is hiring for a Pro­ject Manager

AI al­ign­ment as “nav­i­gat­ing the space of in­tel­li­gent be­havi­our”

Epistemic Arte­facts of (con­cep­tual) AI al­ign­ment research

• Nora_Ammann 5 Aug 2022 18:14 UTC

  LW: 3 AF: 2

  0

  AF

  on: Ro­bust­ness to Scal­ing Down: More Im­por­tant Than I Thought

  Here is another interpretation of what can cause a lack of robustness to scaling down:
  
  (Maybe this is what you have in mind when you talk about single-single alignment not (necessaeraily) scaling to multi-multi alignment—but I am not sure that is the case, and even if it ism I feel pulled to stating it again more as I don’t think it comes out as clearly as I would want it to in the original post.)
  
  Taking the example of an “alignment strategy [that makes] the AI find the preferences of values and humans, and then pursu[e] that”, robustness to scaling down can break if “human values” (as invoked in the example) don’t “survive” reductionism; i.e. if, when we try to apply reducitonism to “human values”, we are left with “less” than what we hoped for.
  
  This is the inverse of saying that there is an important non-linearity when trying to scale (up) from single-single alignment to multi-multi alignment.

  I think interpretation locates the reason for the lack of robustness in neither capabilities nor alignment regime, which is why I wanted to raise it. It’s a claim about the nature or structural properties of “human values”; or a hint that we are deeply confused about human values (e.g. that the term currently refers to an incohernet cluster or “unnatural” abstraction)).
  
  What you say about CEV might capture this fully, but whether it does, I think, is an empirical claim of sorts; a proposed solution to the more general diagnosis that I am trying to propose, namely that (the way we currently use the term) “human values” may itself not be robust to scaling down.
  

• Nora_Ammann 18 Feb 2022 11:13 UTC

  5 points

  on: Monks of Magnitude

  Curious what different aspects the “duration of seclusion” is meant to be a proxy for?
  
  You defindefinitelyitly point at things like “when are they expected to produce intelligible output” and “what sorts of questions appear most relevant to them”. Another dimension that came to mind—but I am not sure you mean or not to include that in the concept—is something like “how often are they allowed/​able to peak directly at the world, relative to the length of periods during which they reason about things in ways that are removed from empirical data”?

• Nora_Ammann 4 Jan 2022 19:36 UTC

  1 point

  on: In­tro­duc­ing the Prin­ci­ples of In­tel­li­gent Be­havi­our in Biolog­i­cal and So­cial Sys­tems (PIBBSS) Fellowship

  PIBBSS Summer Research Fellowship—Q&A event

  • What? Q&A session with the fellowship organizers about the program and application process. You can submit your questions here.

  • For whom? For everyone curious about the fellowship and for those uncertain whether they should apply.

  • When? Wednesday 12th January, 7 pm GMT

  • Where? On Google Meet, add to your calendar

• Nora_Ammann 4 Jan 2022 19:35 UTC

  1 point

  on: An­nounc­ing the PIBBSS Sum­mer Re­search Fellowship

  PIBBSS Summer Research Fellowship—Q&A event

  • What? Q&A session with the fellowship organizers about the program and application process. You can submit your questions here.

  • For whom? For everyone curious about the fellowship and for those uncertain whether they should apply.

  • When? Wednesday 12th January, 7 pm GMT

  • Where? On Google Meet, add to your calendar

[Ex­tended Dead­line: Jan 23rd] An­nounc­ing the PIBBSS Sum­mer Re­search Fellowship

• Nora_Ammann 3 Oct 2021 18:15 UTC

  2 points

  in reply to: jbash’s comment on: The role of tribes in achiev­ing last­ing im­pact and how to cre­ate them

  I think it’s a shame that these days for many people the primary connotation of the word “tribe” is connected to culture wars. In fact, our decision to use this term was in part motivated by wanting to re-appropriate the term to something less politically loaded.
  
  As you can read in our post (see “What is a tribe?”), we mean something particular. As any collective of human beings, it can in principle be subject to excessive in-group/​out-group dynamics but that’s by far not the only, nor the most interesting part of it.

The role of tribes in achiev­ing last­ing im­pact and how to cre­ate them

• Nora_Ammann 12 Jul 2021 8:03 UTC

  5 points

  on: Nora_Am­mann’s Shortform

  Context: (1) Motivations for fostering EA-relevant interdisciplinary research; (2) “domain scanning” and “epistemic translation” as a way of thinking about interdisciplinary research

  [cross-posted to the EA forum in shortform]
  

  List of fields/​questions for interdisciplinary AI alignment research

  The following list of fields and leading questions could be interesting for interdisciplinry AI alignment reserach. I started to compile this list to provide some anchorage for evaluating the value of interdiscplinary research for EA causes, specifically AI alignment.

  Some comments on the list:

  • Some of these domains are likely already very much on the radar of some people, other’s are more speculative.

  • In some cases I have a decent idea of concrete lines of question that might be interesting, in other cases all I do is very broadly gesturing that “something here might be of interest”.

  • I don’t mean this list to be comprehensive or authoritative. On the contrary, this list is definitely skewed by domains I happened to have come across and found myself interested in.

  • While this list is specific to AI alignment (/​safety/​governance), I think the same rationale applies to other EA-relevant domains and I’d be excited for other people to compile similar lists relevant to their area of interest/​expertise.

  Very interested in hearing thoughts on the below!

  Target domain: AI alignment/​safety/​governance

  1. Evolutionary biology

     1. Evolutionary biology seems to have a lot of potentially interesting things to say about AI alignment. Just a few examples include:

        1. The relationship between environment, agent, evolutionary paths (which e.g. relates to to the role of training environments)

        2. Niche construction as an angle on embedded agency

        3. The nature of intelligence

  2. Linguistics and Philosophy of language

     1. Lots of things that are relevant to understanding the nature and origin of (general) intelligence better.

     2. Sub-domains, such as semiotics could, for example, have relevant insights on topics like delegation and interpretability.

  3. Cognitive science and neuroscience

     1. Examples include Minsky’s Society of Minds (“The power of intelligence stems from our vast diversity, not from any single, perfect principle”), Hawkin’s A thousand brains (the role of reference frames for general intelligence), Frinston et al’s Predictive Coding/​Predictive Processing (in its most ambitious versions a near universal theory of all things cognition, perception, comprehension and agency), and many more

  4. Information theory

     1. Information theory is hardly news to the AI alignment idea space. However, there might still be value on the table from deeper dives or more out-of-the-orderly applications of its insights. One example of this might be this paper on The Information Theory of Individuality.

  5. Cybernetics/​Control Systems

     1. Cybernetics seems straightforwardly relevant to AI alignment. Personally, I’d love to have a piece of writing synthesising the most exciting intellectual developments under cybernetics done by someone with awareness of where the AI alignment field is at currently.

  6. Complex systems studies

     1. What does the study of complex systems have to say about robustness, interoperability, emergent alignment? It also offers insights into and methodology for approaching self-organization and collective intelligence which is interesting in particular in multi-multi scenarios.

  7. Heterodox schools of economic thinking

     1. Schools of thought are trying to reimagine the economy/​capitalism and (political) organization, e.g. through decentralization and self-organization, by working on antitrust, by trying to understand potentially radical implications of digitalization on the fabric of the economy, etc. Complexity economics, for example, can help understanding the out-of-equilibrium dynamics that shape much of our economy and lives.

  8. Political economy

     1. An interesting framework for thinking about AI alignment as a socio-technical challenge. Particularly relevant from a multi-multi perspective, or for thinking along the lines of cooperative AI. Pointer: Mapping the Political Economy of Reinforcement Learning Systems: The Case of Autonomous Vehicles

  9. Political theory

     1. The richness of the history of political thought is astonishing; the most obvious might be ideas related to social choice or principles of governance. (A denses while also high-quality overview is offered by this podcast series History Of Ideas.) The crux in making the depth of political thought available and relevant to AI alignment is formalization, which seems extremely undersupplied in current academia for very similar reasons as I’ve argued above.

  10. Management and organizational theory, Institutional economics and Institutional design

      1. Has things to say about e.g. interfaces (read this to get a gist for why I think interfaces are interesting for AI alignment); delegation ( e.g. Organizations and Markets by Herbert SImon; (potentially) the ontology form forms and (the relevant) agent boundaries (e.g. The secret to social forms has been in institutional economics all along?)

      2. Talks for example about desiderata for institutions like robustness (e.g. here), or about how to understand and deal with institutional path-dependencies (e.g. here).