Research Scientist at DeepMind. Creator of the Alignment Newsletter. http://rohinshah.com/
rohinmshah(Rohin Shah)
Karma: 9,616
I’m assuming listeners will only do things if they don’t mind doing them, i.e. that my words won’t coerce people,
I feel like this assumption seems false. I do predict that (at least in the world where we didn’t have this discussion) your statement would create a social expectation for the people to report true, relevant facts, and that this social expectation would in fact move people in the direction of reporting true, relevant facts.
I immediately made the inference myself on reading your comment. There was no choice in the matter, no execution of a deliberate strategy on my part, just an inference that Anna wants people to give the facts, and doesn’t think that fear of reprisal is particularly important to care about. Well, probably, it’s hard to remember exactly what I thought, but I think it was something like this. I then thought about why this might be, and how I might have misunderstood. In hindsight, the explanation you gave above should have occurred to me, that is the sort of thing that people who speak literally would do, but it did not.
I think there are lots of LWers who, like me, make these sorts of inferences automatically. (And I note that these kinds of inferences are excellent for believing true things about people outside of LW.) I think this is especially true of people in the same reference class as Zoe, and that such people will feel particularly pressured by it. (There are a sadly large number of people in this community who have a lot of self-doubt / not much self-esteem, and are especially likely to take other people’s opinions seriously, and as a reason for them to change their behavior even if they don’t understand why.) This applies to both facts that are politically-pro-Leverage and facts that are politically-anti-Leverage.
So overall, yes, I think your words would lead people to infer that it would be better for them to report true relevant facts and that any fear they have is somehow misplaced, and to be pressured by that inference into actually doing so, i.e. it coerces them.
I don’t have a candidate alternative norm. (I generally don’t think very much about norms, and if I made one up now I’m sure it would be bad.) But if I wanted to convey something similar in this particular situation, I would have said something like “I would love to know additional true relevant facts, but I recognize there is a risk that others will take them in a politicized way, or will use them as an excuse to falsely judge you, so please only do this if you think the benefits are worth it”.
(Possibly this is missing something you wanted to convey, e.g. you wish that the community were such that people didn’t have to fear political judgment?)
(I also agree with TekhneMakre’s response about authority.)
Hypothesis 2 feels truer than hypothesis 1.
(Just to state the obvious: it is clearly not as bad as the words “coercion” and “gaslighting” would usually imply. I am endorsing the mechanism, not the magnitude-of-badness.)
I agree that hypothesis 1 could be an underlying generator of why the effect in hypothesis 2 exists.
I think I am more confident in the prediction that these sorts of statements do influence people in ways-I-don’t-endorse, than in any specific mechanism by which that happens.
It sounds like you are predicting that the people who are sharing true relevant facts have values such that the long-term benefits to the group overall outweigh the short-term costs to them. In particular, it’s a prediction about their values (alongside a prediction of what the short-term and long-term effects are).
I’ll just note that, on my view of the short-term and long-term effects, it seems pretty unclear whether by my values I should share additional true relevant information, and I lean towards it being negative. Of course, you have way more private info than me, so perhaps you just know a lot more about the short-term and long-term effects.
I’m also not a fan of requests that presume that the listener is altruistic, and willing to accept personal harm for group benefit. I’m not sure if that’s what you meant—maybe you think in the long term sharing of additional facts would help them personally, not just help the group.
Fwiw I don’t have any particularly relevant facts. I once tagged along with a friend to a party that I later (i.e. during or after the party) found out was at Leverage. I’ve occasionally talked briefly with people who worked at Leverage / Paradigm / Reserve. I might have once stopped by a poster they had at EA Global. I don’t think there have been other direct interactions with them.
I agree that’s possible, but then I’d say something like “I would love to know additional true relevant facts, but I recognize there are real risks to this and only recommend people do this if they think the benefits are worth it”.
Analogy: it could be worth it for an employee to publicly talk about the flaws of their company / manager (e.g. because then others know not to look for jobs at that company), even though it might get them fired. In such a situation I would say something like “It would be particularly helpful to know about the flaws of company X, but I recognize there are substantial risks involved and only recommend people do this if they feel up to it”. I would not say “I hope people don’t refrain from speaking up about the flaws of company X out of fear that they might be fired”, unless I had good reason to believe they wouldn’t be fired, or good reason to believe that it would be worth it on their values (though in that case presumably they’d speak up anyway).
Refraining from sharing true relevant facts, out of fear that others will take them in a politicized way, or will use them as an excuse for false judgments.
Are you somehow guaranteeing or confidently predicting that others will not take them in a politicized way, use them as an excuse for false judgments, or otherwise cause harm to those sharing the true relevant facts? If not, why are you asking people not to refrain from sharing such facts?
(My impression is that it is sheer optimism, bordering on wishful thinking, to expect such a thing, that those who have such a fear are correct to have such a fear, and so I am confused that you are requesting it anyway.)
Yeah, I agree the statement is false as I literally wrote it, though what I meant was that you could easily believe you are in the kind of simulation where there is no extraordinary impact to have.
Edited to
Selection theorems are helpful because (1) they can provide additional assumptions that can help with learning human values
and
[...] the resulting agents can be represented as maximizing expected utility, if the agents don’t have internal state.
(For the second one, that’s one of the reasons why I had the weasel word “could”, but on reflection it’s worth calling out explicitly given I mention it in the previous sentence.)
Thanks for this and the response to my other comment, I understand where you’re coming from a lot better now. (Really I should have figured it out myself, on the basis of this post.) New summary:
This post proposes a research area for understanding agents: **selection theorems**. A selection theorem is a theorem that tells us something about agents that will be selected for in a broad class of environments. Selection theorems are helpful because (1) they can provide additional assumptions that can help with learning values by observing human behavior, and (2) they can tell us likely properties of the agents we build by accident (think inner alignment concerns).
As an example, [coherence arguments](https://www.alignmentforum.org/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities) demonstrate that when an environment presents an agent with “bets” or “lotteries”, where the agent cares only about the outcomes of the bets, then any “good” agent can be represented as maximizing expected utility. (What does it mean to be “good”? This can vary, but one example would be that the agent is not subject to Dutch books, i.e. situations in which it is guaranteed to lose resources.) This can then be turned into a selection argument by combining it with something that selects for “good” agents. For example, evolution will select for agents that don’t lose resources for no gain, so humans are likely to be represented as maximizing expected utility. Unfortunately, many coherence arguments implicitly assume that the agent has no internal state, which is not true for humans, so this argument does not clearly work. As another example, our ML training procedures will likely also select for agents that don’t waste resources, which could allow us to conclude that the resulting agents can be represented as maximizing expected utility.
Coherence arguments aren’t the only kind of selection theorem. The <@good(er) regulator theorem@>(@Fixing The Good Regulator Theorem@) provides a set of scenarios under which agents learn an internal “world model”. The [Kelly criterion](http://www.eecs.harvard.edu/cs286r/courses/fall10/papers/Chapter6.pdf) tells us about scenarios in which the best (most selected) agents will make bets as though they are maximizing expected log money. These and other examples are described in [this followup post](https://www.alignmentforum.org/posts/N2NebPD78ioyWHhNm/some-existing-selection-theorems).
The rest of this post elaborates on the various parts of a selection theorem, and provides advice on how to make original research contributions in the area of selection theorems. Another [followup post](https://www.alignmentforum.org/posts/RuDD3aQWLDSb4eTXP/what-selection-theorems-do-we-expect-want) describes some useful properties for which the author expects there are useful selections theorems to prove.
New opinion:
People sometimes expect me to be against this sort of work, because I wrote <@Coherence arguments do not imply goal-directed behavior@>. This is not true. My point in that post is that coherence arguments _alone_ are not enough, you need to combine them with some other assumption (for example, that there exists some “resource” over which the agent has no terminal preferences). I do think it is plausible that this research agenda gives us a better picture of agency that tells us something about how AI systems will behave, or something about how to better infer human values. While I am personally more excited about studying particular development paths to AGI rather than more abstract agent models, I do think this research would be more useful than other types of alignment research I have seen proposed.
Planned summary for the Alignment Newsletter:
This post proposes a research area for understanding agents: **selection theorems**. A selection theorem is a theorem that tells us something about agents that will be selected for in a broad class of environments. Selection theorems are helpful because they tell us likely properties of the agents we build.
As an example, [coherence arguments](https://www.alignmentforum.org/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities) demonstrate that when an environment presents an agent with “bets” or “lotteries”, where the agent cares only about the outcomes of the bets, then any non-dominated agent can be represented as maximizing expected utility. (What does it mean to be non-dominated? This can vary, but one example would be that the agent is not subject to Dutch books, i.e. situations in which it is guaranteed to lose money.) If you combine this with the very reasonable assumption that we will tend to build non-dominated agents, then we can conclude that we select for agents that can be represented as maximizing expected utility.
Coherence arguments aren’t the only kind of selection theorem. The <@good(er) regulator theorem@>(@Fixing The Good Regulator Theorem@) provides a set of scenarios under which agents learn an internal “world model”. The [Kelly criterion](http://www.eecs.harvard.edu/cs286r/courses/fall10/papers/Chapter6.pdf) tells us about scenarios in which the best (most selected) agents will make bets as though they are maximizing expected log money. These and other examples are described in [this followup post](https://www.alignmentforum.org/posts/N2NebPD78ioyWHhNm/some-existing-selection-theorems).
The rest of this post elaborates on the various parts of a selection theorem, and provides advice on how to make original research contributions in the area of selection theorems. Another [followup post](https://www.alignmentforum.org/posts/RuDD3aQWLDSb4eTXP/what-selection-theorems-do-we-expect-want) describes some useful properties for which the author expects there are useful selections theorems to prove.
Planned opinion:
People sometimes expect me to be against this sort of work, because I wrote <@Coherence arguments do not imply goal-directed behavior@>. This is not true. My point in that post is that coherence arguments _alone_ are not enough, you need to combine them with some other assumption (for example, that there is a money-like resource over which the agent has no terminal preferences). Similarly, I don’t expect this research agenda to find a selection theorem that says that an existential catastrophe occurs _assuming only that the agent is intelligent_, but I do think it is plausible that this research agenda gives us a better picture of agency that tells us something about how AI systems will behave, because we think the assumptions involved in the theorems are quite likely to hold. While I am personally more excited about studying particular development paths to AGI rather than more abstract agent models, I would not actively discourage anyone from doing this sort of research, and I think it would be more useful than other types of research I have seen proposed.
At the same time, better Selection Theorems directly tackle the core conceptual problems of alignment and agency; I expect sufficiently-good Selection Theorems would get us most of the way to solving the hardest parts of alignment.
The former statement makes sense, but can you elaborate on the latter statement? I suppose I could imagine selection theorems revealing that we really do get alignment by default, but I don’t see how they quickly lead to solutions to AI alignment if there is a problem to solve.
ASSUMPTION 1: There’s a “secret sauce” of human intelligence, and it looks like a learning algorithm (and associated inference algorithm).
ASSUMPTION 2: It’s a fundamentally different learning algorithm from deep neural networks. I don’t just mean a different neural network architecture, regularizer, etc. I mean really different, like “involving probabilistic program inference algorithms” or whatever.
ASSUMPTION 3: The algorithm is human-legible, but nobody knows how it works yet.
ASSUMPTION 4: We’ll eventually figure out this “secret sauce” and get Transformative AI (TAI).
These seem easily like the load-bearing part of the argument; I agree the stuff you listed follows from these assumptions but why should these assumptions be true?
I can imagine justifying assumption 2, and maybe also assumption 1, using biology knowledge that I don’t have. I don’t see how you justify assumptions 3 and 4. Note that assumption 4 also needs to include a claim that we figure out the “secret sauce” sooner than other paths to AGI, despite lots of effort being put into them already.
Hmm, you might want to reach out to CHAI folks, though I don’t have a specific person in mind at the moment. (I myself am working on different things now.)
[AN #166]: Is it crazy to claim we’re in the most important century?
Another problem with using GDP to predict something like “continuously increasing impact on the world” is that it seems like new technologies often lead to huge surplus that wouldn’t get captured in a GDP metric. Search engines are ridiculously useful, people say they would pay a lot to not lose them, and yet they’re free.
(This is arguably the same problem as you identify, in that as you mention you have to quantify the value of a search engine or a smartphone to include it in a GDP metric, and there isn’t an obvious way to assign a value in 1960-dollars to a search engine, so you just go with what people pay for it now.)
Personally, the main evidence I rely on for “no discontinuities in impact” is that it seems like across a range of industries / technologies, after the first zero-to-one transition in which the technology is invented, the improvements on the technology tend to be continuous / incremental, and so too is its impact on the world.
(This needs to be combined with a claim that the zero-to-one transition for AI will lead to AI systems that are subhuman and so not very impactful. My impression is that some people disagree with this, seeing current AI systems as qualitatively-different-from-AGI, and expecting a completely different zero-to-one transition to AGI, in which the resulting AGI is immediately much better than humans on some important axis, like ability to self-improve. I’m not sure why they think this, if in fact this is an accurate representation of their beliefs.)
Planned summary for the Alignment Newsletter:
This post summarizes several AI takeover scenarios that have been proposed, and categorizes them according to three main variables. **Speed** refers to the question of whether there is a sudden jump in AI capabilities. **Uni/multipolarity** asks whether a single AI system takes over, or many. **Alignment** asks what goals the AI systems pursue, and if they are misaligned, further asks whether they are outer or inner misaligned. They also analyze other properties of the scenarios, such as how agentic, general and/or homogenous the AI systems are, and whether AI systems coordinate with each other or not. A [followup post](https://www.alignmentforum.org/posts/zkF9PNSyDKusoyLkP/investigating-ai-takeover-scenarios) investigates social, economic, and technological characteristics of these scenarios. It also generates new scenarios by varying some of these factors.
Since these posts are themselves summaries and comparisons of previously proposed scenarios that we’ve covered in this newsletter, I won’t summarize them here, but I do recommend them for an overview of AI takeover scenarios.
Planned summary for the Alignment Newsletter:
It has been claimed that there’s no fire alarm for AGI, that is, there will be no specific moment or event at which AGI risk becomes sufficiently obvious and agreed upon, so that freaking out about AGI becomes socially acceptable rather than embarrassing. People often implicitly argue for waiting for an (unspecified) future event that tells us AGI is near, after which everyone will know that it’s okay to work on AGI alignment. This seems particularly bad if no such future event (i.e. fire alarm) exists.
This post argues that this is not in fact the implicit strategy that people typically use to evaluate and respond to risks. In particular, it is too discrete. Instead, people perform “the normal dance of accumulating evidence and escalating discussion and brave people calling the problem early and eating the potential embarrassment”. As a result, the existence of a “fire alarm” is not particularly important.
Note that the author does agree that there is some important bias at play here. The original fire alarm post is implicitly considering a _fear shame hypothesis_: people tend to be less cautious in public, because they expect to be negatively judged for looking scared. The author ends up concluding that there is something broader going on and proposes a few possibilities, many of which still suggest that people will tend to be less cautious around risks when they are observed.
Some points made in the very detailed, 15,000-word article:
1. Literal fire alarms don’t work by creating common knowledge, or by providing evidence of a fire. People frequently ignore fire alarms. In one experiment, participants continued to fill out questionnaires while a fire alarm rang, often assuming that someone will lead them outside if it is important.
2. They probably instead work by a variety of mechanisms, some of which are related to the fear shame hypothesis. Sometimes they provide objective evidence that is easier to use as a justification for caution than a personal guess. Sometimes they act as an excuse for cautious or fearful people to leave, without the implication that those people are afraid. Sometimes they act as a source of authority for a course of action (leaving the building).
3. Most of these mechanisms are amenable to partial or incremental effects, and in particular can happen with AGI risk. There are many people who have already boldly claimed that AGI risk is a problem. There exists person-independent evidence; for example, surveys of AI researchers suggest a 5% chance of extinction.
4. For other risks, there does not seem to have been a single discrete moment at which it became acceptable to worry about them (i.e. no “fire alarm”). This includes risks where there has been a lot of caution, such as climate change, the ozone hole, recombinant DNA, COVID, and nuclear weapons.
5. We could think about _building_ fire alarms; many of the mechanisms above are social ones rather than empirical facts about the world. This could be one out of many strategies that we employ against the general bias towards incaution (the post suggests 16).
Planned opinion:
I enjoyed this article quite a lot; it is _really_ thorough. I do see a lot of my own work as pushing on some of these more incremental methods for increasing caution, though I think of it more as a combination of generating more or better evidence, and communicating arguments in a manner more suited to a particular audience. Perhaps I will think of new strategies that aim to reduce fear shame instead.
Some notes from a 2018 CHAI meeting on this topic (with some editing). I don’t endorse everything on here, nor would CHAI-the-organization.
Learning part of your model that previously was fixed.
Can be done using neural nets, other ML models, or uncertainty (probability distributions)
Example: learning biases instead of hardcoding Boltzmann rationality
Relatedly, treating an object as evidence instead of as ground truth
Example: Inverse Reward Design treats the specified reward as evidence about the true reward
Looking at current examples of the problem and/or its solutions:
How does the human brain / nature do it?
How does human culture/society do this
How has cognitive science formalized similar problems / what insights has it produced that we can build on?
Principal-agent models/Contracting theory
Adversarial agents for robustness
Internal design of the system to be adversarial inherently (e.g. debate)
External use of adversaries for testing: red teaming, adversarial training
‘Normative Bandwidth’
How much information about the correct behavior is actually conveyed vs how much information does the robot policy assume is conveyed
E.g. reward functions that are interpreted literally means that you are getting all information necessary for getting the optimal policy—that’s a huge amount of information, that assumption is always wrong. What’s actually conveyed is a much smaller amount of information—something like what Inverse Reward Design does (where it says the reward function only conveys information about good behavior in the training environments).
Proactive Learning (i.e. what if we ask the human?)
Induction (see e.g. iterated amplification)
Get good safety properties in simple situations, and then use them to build something more capable while preserving safety properties
Analyze a simple model of the situation in theory
Indexical uncertainty (uncertainty about your identity)
Rationality—either make sure the agent is rational, or make sure it isn’t (i.e. don’t build agents)
Thinking about the human-robot system as a whole, rather than the robot in isolation. (See e.g. CIRL / assistance games.)
How would you do it with infinite resources (relaxed constraints)?
E.g. AIXI, Solomonoff induction, open-source game theory
I imagine Rohin Shah would see a proposed solution and ask how it intervenes on a threat model (also correct me if I’m wrong!).
That’s certainly one heuristic I often use :)
Sounds like a cool project, I’m looking forward to seeing the results!
You’re reading too much into my response. I didn’t claim that Anna should have this extra onus. I made an incorrect inference, was confused, asked for clarification, was still confused by the first response (honestly I’m still confused by that response), understood after the second response, and then explained what I would have said if I were in her place when she asked about norms.
(Yes, I do in fact think that the specific thing said had negative consequences. Yes, this belief shows in my comments. But I didn’t say that Anna was wrong/bad for saying the specific thing, nor did I say that she “should” have done something else. Assuming for the moment that the specific statement did have negative consequences, what should I have done instead?)
(On the actual question, I mostly agree that we probably have too many demands on public communication, such that much less public communication happens than would be good.)
I also would have been fine with “I hope people share additional true, relevant facts”. The specific phrasing seemed bad because it seemed to me to imply that the fear of reprisal was wrong. See also here.