Research Scientist at DeepMind. Creator of the Alignment Newsletter. http://rohinshah.com/
rohinmshah(Rohin Shah)
Karma: 10,131
[AN #171]: Disagreements between alignment “optimists” and “pessimists”
To be clear, I didn’t mean this comment as “stop cricitizing me”. I meant it as “I think the statement is factually incorrect”. The reason that the newsletter has more ML in it than MIRI work is just that there’s more (public) work produced on the ML side.
I don’t think it’s about the lack of papers, unless by papers you mean the broader category of “public work that’s optimized for communication”.
Alignment Newsletter, ect. are naturally going to strongly focus on ML-style research and might not even have the capability to vet MIRI-style research.
I think I’ve summarized ~every high-effort public thing from MIRI in recent years (I’m still working on the late 2021 conversations). I also think I understood them better (at time of summarizing) than most other non-MIRI people who have engaged with it.
MIRI also has a standing offer from me to work with me to produce summaries of new things they think should have summaries (though they might have forgotten it at this point—after they switched to nondisclosed-by-default research I didn’t bother reminding them).
I broadly agree with the things you’re saying; I think it mostly comes down to the actual numbers we’d assign.
This seems really extreme, if I’m not misunderstanding you. (My own number is like 1x-5x.) Assuming your intent alignment risk is 10%, your AI persuasion risk is only 1/1000?
Yeah, that’s about right. I’d note that it isn’t totally clear what the absolute risk number is meant to capture—one operationalization is that it is P(existential catastrophe occurs, and if we had solved AI persuasion but the world was otherwise exactly the same, then no existential catastrophe occurs) -- I realize I didn’t say exactly this above but that’s the one that is mutually exhaustive across risks, and the one that determines expected value of solving the problem.
To justify the absolute number of 1/1000, I’d note that:
The case seems pretty speculative + conjunctive—you need people to choose to use AI to be very persuasive (instead of, idk, retiring to live in luxury in small insular subcommunities), you’d need the AI to be better at persuasion than defending against persuasion (or for people to choose not to defend), and you’d need this to be so bad that it leads to an existential catastrophe.
I feel like if I talked to lots of people the amount I’ve talked with you / others about AI persuasion (i.e. not very much, but enough to convey a basic idea) I’d end up having 10-300 other risks of similar magnitude and plausibility. Under the operationalization I gave above, these probabilities would be mutually exclusive. So that places an upper bound of 1⁄300 − 1⁄10 on any given problem.
I don’t expect this bound to be tight. For example, if it were tight, that would imply that existential catastrophe is guaranteed. But more importantly, there are lots of worlds in which existential catastrophe is overdetermined because society is terrible at coordinating. If you condition on “existential catastrophe” and “AI persuasion was a big problem”, I update that we were really bad at coordination and so I also think that there would be lots of other problems such that solving persuasion wouldn’t prevent the existential catastrophe. (Whereas alignment feels much more like a direct technical challenge—while there certainly is an update against societal coordination if we get an existential catastrophe with a failure of alignment, the update seems a lot smaller, and so I’m more optimistic that solving alignment means that the existential catastrophe doesn’t happen at all.)
The 100x differential between alignment and persuasion comes mostly because points (2) and (3) above don’t apply to alignment, point (1) applies only in part—given my state of knowledge, the case for alignment failure seems much less speculative (though obviously still speculative), though it is still quite conjunctive.
I see, but I think at least part of the problem with threats is that I’m not sure what I care about, which greatly increases my “attack surface”. For example, if I knew that negative utilitarianism is definitely wrong, then threats to torture some large number of simulated people wouldn’t be effective on me (e.g., under total utilitarianism, I could use the resources demanded by the attacker to create more than enough happy people to counterbalance whatever they threaten to do).
That’s a fair point, I agree this is a way in which full knowledge of human values can help avoid potentially significant risks in a way that intent alignment doesn’t.
I’m not sure I understand the distinction that you’re drawing here. (It seems like my scenarios could also be interpreted as failures where AI don’t know enough human values, or maybe where humans themselves don’t know enough human values.) What are some examples of what your claim was about?
Examples:
Your AI thinks it’s acceptable to inject you with heroin, because it predicts you will then want more heroin.
Your AI is uncertain whether you’d prefer to explore space or stay on Earth. It randomly guesses that you want to stay on Earth and takes irreversible actions on your behalf that force you to stay on Earth.
In contrast, something like a threat doesn’t count, because you know that the outcome if the threat is executed is not something you want; the problem comes because you don’t know how to act in a way that both disincentivizes threats and also doesn’t lead to (too many) threats being enforced. In particular, the problem is not that you don’t know which outcomes are bad.
As in, the total expected value lost through such scenarios isn’t as large as the expected value lost through the risk of failing to solve intent alignment?
No, the expected value of marginal effort aimed at solving these problems isn’t as large as the expected value of marginal effort on intent alignment.
(I don’t like talking about “expected value lost” because it’s not always clear what does and doesn’t count as part of that. For example I think it’s nearly inevitable that different people will have different goals and so the future will not be exactly as any one of them desired; should I say that there’s a lot of expected value lost from “coordination problems” for that reason? It seems a bit weird to say that if you think there isn’t a way to regain that “expected value”.)
Can you give some ballpark figures of how you see each side of this inequality?
Uh, idk. It’s not something I have numbers on. But I suppose I can try and make up some very fake numbers for, say, AI persuasion. (Before I actually do the exercise, let me note that I could imagine the exercise coming out with numbers that favor persuasion over intent alignment; this probably won’t change my mind and would instead make me distrust the numbers, but I’ll publish them anyway.)
To change an existentially bad outcome from AI persuasion, I’d imagine first figuring out some solutions, then figuring out how to implement them, and then getting the appropriate people to implement them; seems like you need all of these steps in order to make any difference. (Could be technical or governance though.) It feels especially hard to do so at the moment, given how little we know about future AI capabilities and when each potential capability arrives. Maybe:
Alignment is 100x more likely to be an existentially risky problem at all (think of this as the ratio between probabilities of existential catastrophe by the given problem assuming no intervention from longtermists).
A piece of alignment work now is 10x more likely to target the right problem than a similar piece of work for persuasion.
A piece of alignment work now is currently 10x harder to produce than a similar piece of work for persuasion.
So I guess overall I’m at ~100x, very very roughly?
How? How would you train an AI to distinguish between philosophy optimized for persuasion, and correct or well-intentioned philosophy that just happens to be very persuasive?
Putting on my “what would I do” hat, I’m imagining that the AI doesn’t know that it was specifically optimized to be persuasive, but it does know that there are other persuasive counterarguments that aren’t being presented, and so it says that it looks one-sided and you might want to look at these other counterarguments. Or it says that there are other counterfactual letters you could have received, such that after you read them you’d be convinced of the opposite position, and then it asks whether you still want to read the letter.
If your AI doesn’t know about the counterarguments, and the letter is persuasive even to the AI, then it seems like you’re hosed, but I’m not sure why to expect that.
A lot of people are playing status games where faith/loyalty to their cause/ideology gains them a lot of status points. Why wouldn’t they ask their AI for help with this? Or do you imagine them asking for something like “more faith”, but AIs understand human values well enough to not interpret that as “lock in values”?
I totally expect them to ask AI for help with such games. I don’t expect (most of) them to lock in their values such that they can’t change their mind.
I’m not entirely sure why you do expect this. Maybe you’re viewing them as consciously optimizing for winning status games + a claim that the optimal policy is to lock in your values? But what if the values rewarded by the status games (as they already seem to, e.g. moving from atheism to anti-racism)? In that case it seems like you don’t want to lock in your values, to better play the status game in the future.
The former seems to just require that the AI is good at reasoning about mathematical/empirical matters (e.g., are there many simulations of me actually being run in some universe or set of universes) which I think AIs will be good at by default, whereas dealing with the threats seems to require reasoning about hard philosophical problems like decision theory and morality.
If you’ve determined a set of universes to care about, then shouldn’t at least decision theory reduce to mathematical / empirical matters about which decision procedure gets the most value across the set of universes? I do agree that moral questions are still not mathematical / empirical, but I don’t find that all that persuasive. I expect AIs will be able to do the sort of philosophical reasoning that we do, and the question of whether we should care about simulations seems way way easier than the question about which simulations of me are being run, by whom, and what they want.
I’m guessing you feel better about mathematical / empirical reasoning because there’s a ground truth that says when that reasoning is done well. I don’t particularly find the existence of a ground truth to be all that big a deal—it probably helps but doesn’t seem tremendously important.
I expect that AIs (or humans) who are less cautious or who think their values can be easily expressed as utility functions will do this first, and thereby gain an advantage over everyone else and maybe forcing them to follow.
Fair enough—I agree this is plausible (though only plausible; it doesn’t seem like we’ve sacrificed everything to Moloch yet).
I don’t think it’s so much that the coordination involving humans is a lot of work, but rather that we don’t know how to do it in a way that doesn’t cause a lot of waste, similar to a democratically elected administration implementing a bunch of policies only to be reversed by the next administration that takes power, or lawmakers pursuing pork barrel projects that collectively make almost everyone worse off, or being unable to establish and implement easy policies (see COVID again). (You may well have something in mind that works well in the context of intent aligned AI, but I have a prior that says this class of problems is very difficult in general so I’d need to see more details before I update.)
If we’re imagining coordination between a billion AIs, that seems less obviously doable. I still think that, if you’ve solved intent alignment, it seems much easier. Democratic elections are not just about coordination—they’re also about alignment, since politicians have to optimize for being re-elected. It seems like you could do a lot better if you didn’t have to worry about the alignment part.
To be clear, my original claim was for hypothetical scenarios where the failure occurs because the AI didn’t know human values, rather than cases where the AI knows what the human would want but still a failure occurs. (I didn’t state this explicitly because I was replying to the post, which focuses specifically on the problem of not knowing all of human values.) I think most of your failures are of the latter type, and I wouldn’t make a similar claim for such failures—they seem plausible and worth attention. I do still think they are not as important as intent alignment. Talking about each one individually:
I think unless we make sure the AI can distinguish between “correct philosophy” or “well-intentioned philosophy” and “philosophy optimized for persuasion”, each human will become either compromised (if they’re not very cautious and read such messages) or isolated from the rest of humanity with regard to philosophical discussion (if they are cautious and discard such messages). This doesn’t seem like an ok outcome to me. Can you explain more why you aren’t worried?
Mostly I’d hope that AI can tell what philosophy is optimized for persuasion, or at least is capable of presenting counterarguments persuasively as well. But if your AI can’t even tell what is persuasive then you’re in trouble, but I’m not sure why to expect that.
I can imagine that if you subscribe to a metaethics in which a person can’t be wrong about morality (i.e., some version of anti-realism), then you might think it’s fine to lock in whatever values one currently thinks they ought to have. Is this your reason for “seems fine”, or something else?
No, it just doesn’t seem hugely terrible for a few people to lock in bad values, as long as the vast majority do not. (And I don’t expect a large number of people to explicitly try to lock in their values.)
If the AI isn’t very good at dealing with “exotic philosophical cases” then it’s not going to be of much help with this problem, and a lot of humans (including me) probably aren’t very good at thinking about this either, so we probably end up with a lot of humans succumbing to such acausal attacks.
It seems odd to me that it’s sufficiently competent to successfully reason about simulations enough that an acausal threat can actually be made, but then not competent at reasoning about exotic philosophical cases, and I don’t particularly expect this to happen.
Or some other reason to think that AIs that are aligned with different humans will find ways to cooperate (as efficiently as merging utility functions probably will be), without either a full understanding of human values or risking permanent loss of some parts of their complex values?
I don’t expect AIs to have clean crisp utility functions of the form “maximize paperclips” (at least initially).
If you’re using some form of indirect normativity (which is the general approach I’m excited about), then it seems like you could have an agreement between to AIs about how to aggregate / merge these two definitions of “the good” for different humans (e.g. each human gets “points” in proportion to their current resources, and then they can spend points to have influence over specific choices; presumably you could do better but this seems like a fine start). I expect this to be way less work than the complicated plans that the AI is enacting, so it isn’t a huge competitiveness hit.
Agreed that’s a possible good outcome, but seems far from a sure thing. Such laws would have to more intrusive than anything people are currently used to, since attackers can create simulated suffering within the “privacy” of their own computers or minds. I suppose if such threats become a serious problem that causes a lot of damage, people might agree to trade off their privacy for security. The law might then constitute a risk in itself, as the implementation mechanism might be subverted/misused to create a form of totalitarianism.
I agree it is far from a sure thing.
Your AI should tell you that it’s worried about your friend being compromised, make sure you have an understanding of the consequences, and then go with your decision.
Seems fine. Maybe your AI warns you about the risks before helping.
Seems like an important threat that you (and your AI) should try to resolve.
If you mean a utility function over universe histories (as opposed to e.g. utility functions over some finite set of high-level outcomes) this seems pretty rough. Mostly I would hope that this situation doesn’t arise, because none of the humans can come up with utility functions in this way, and the AIs that are aligned with humans have other ways of cooperating that don’t require eliciting a utility function over universe histories.
Idk, seems pretty unclear, but I’d hope that these situations can’t come up thanks to laws that prevent people from enforcing such threats.
In addition to
(1) isn’t likely to come up, or (2) can be solved by deferring to the human, or (3) doesn’t matter very much.
I should probably add a fourth:
(4) can be solved through governance (laws, regulations, norms, etc)
Fwiw, in this story I find myself surprised at aAlice’s confidence in her theory. If I were telling a story about an unlearnable theory of mind, I’d be leaning on huge uncertainty that prevents aAlice from doing anything.
I find “large numbers of low-impact discoveries” to be the most compelling, though I’d also add:
Variance becomes lower as people put more effort into finding discoveries (intuitively, each person has some chance of finding a discovery, all of which are approximately independent from each other).
There can be high-impact discoveries, but those are found earlier (partly because people are looking for them, and partly because there are more ways to stumble upon high-impact discoveries).
Very cool!
A note of caution: when I handcoded weights of a neural network (in my case, to solve a gridworld RL problem), I was able to encode the optimal policy—but the algorithm that was later learned by gradient descent was very different. Partly this was because I only required myself to produce the right action, so I often had the (equivalent of) Q-values for different actions be very very close to each other, whereas the neural network ended up having Q-values that were further apart from each other, which was incentivized by the loss function even though it didn’t make a difference to the optimal policy.
So to the extent you’re trying to learn what a neural net trained by gradient descent would do, I’d recommend that you spend some time looking at the trained neural net to see whether it is using a similar sort of algorithm as the one you’re implementing.
In my model, one should be deeply skeptical whenever the answer to ‘what would do the most good?’ is ‘get people like me more money and/or access to power.’ One should be only somewhat less skeptical when the answer is ‘make there be more people like me’ or ‘build and fund a community of people like me.’
[...]
Lightcone Infrastructure
[...]
if you gave that person [...] a credit card and told me ‘seriously, use this for whatever you want as long as it’s putting you in a better position to do the stuff worth doing, it doesn’t matter, stop caring about money unless it’s huge amounts.’
Seems like these are pretty centrally “build and fund a community of people like me”. Do you agree with this, and think that you have enough evidence to overcome the default skepticism? Or do you think Lightcone doesn’t actually fall into that reference class?
My guess is that it’s the first one, and that you’d say that unlike the vast majority of other orgs, Lightcone is clearly composed of people who are Doing Thing and will lead to other people Doing Thing better, and this isn’t true of most other things where you have default skepticism?
Apparently this has been nominated for the review. I assume that this is implicitly a nomination for the book, rather than my summary of it. If so, I think the post itself serves as a review of the book, and I continue to stand by the claims within.
I think 24% for “there will be a big discontinuity at some point in the history of a field” is pretty reasonable, though I have some quibbles with your estimates (detailed below). I think there are a bunch of additional facts that make me go a lot lower than that on the specific question we have with AI:
We’re talking about a discontinuity at a specific moment along the curve—not just “there will be a discontinuity in AI progress at some point”, but specifically “there will be a discontinuity around the point where AI systems first reach approximately human-level intelligence”. Assigning 5% to a discontinuity at a specific region of the curve can easily be compatible with 24% of a discontinuity overall.
We also know that the field of AI has been around for 60 years and that there is a lot of effort being put into building powerful AI systems. I expect that the more effort is being put into something, the more the low-hanging fruit / “secrets” are already plucked, and the less likely discontinuities are.
We can’t currently point to anything that seems like it should cause a discontinuity in the future. It seems to me like for many of the “physics hack” style of discontinuity, the discontinuity would have been predictable in advance (I’m thinking especially of nukes and spaceflight here). Though possibly this is just hindsight bias on my part.
We were talking about a huge discontinuity—that the first time we destroy cities, we will also destroy the Earth. (And I think we’re talking about a similarly large discontinuity in the AI case, though I’m not actually sure.) These intuitively feel way larger than your big discontinuities. Though as a counterpoint, I also think AI will be a way bigger deal than most of the other technologies, so a similar discontinuity in some underlying trend could lead to a much bigger discontinuity in terms of impact. (Still, if we talk about “smaller” discontinuities like 1-year doubling of GDP before 4-year doubling of GDP, I put more probability on it, relative to something like “the world looks pretty similar to today’s world, and then everyone drops dead”.)
(All of these together would push me way lower than 5%, if ignoring model uncertainty / “maybe I’m wrong” + noting that the future is hard to predict.)
It seems to me that the biggest point of disagreement is on (3), and this is why in the conversation I keep coming back to
My impression is that Eliezer thinks that “general intelligence” is a qualitatively different sort of thing than that-which-neural-nets-are-doing, and maybe that’s what’s analogous to “entirely new physics”. I’m pretty unconvinced of this, but something in this genre feels quite crux-y for me.
I do think “look at historical examples” is a good thing to do, so I’ll go through each of your discontinuities in turn. Note that I know very little about most of these areas and haven’t even read the Wikipedia page for most of them, so lots of things I say could be completely wrong:
Aviation: I assume you’re talking about the zero-to-one discontinuity from “no flight” to “flight”? I do agree that we’ll see zero-to-one discontinuities on particular AI capabilities, e.g. language models learn to do arithmetic quite discontinuously. This seems pretty irrelevant to the case with AI. (Notably, the Wright flyer didn’t have much of an impact on things people cared about, and not that many people were working on flight to my knowledge.)
Nukes: Agree that this is a zero-to-one discontinuity from “physics hack” (but note that it did involve a huge amount of effort). Unlike the Wright flyer, it did have a huge impact on things people cared about.
Petroleum: Not sure what the discontinuity is—is it that “amount of petroleum used” increased discontinuously? If so, I very much expect such discontinuities to happen; they’ll happen any time a better technology replaces a worse technology. (Put another way, the reason to expect continuous progress is that there is optimization pressure on the metric and so the low-hanging fruit / “secrets” have already been taken; there wouldn’t have been much optimization pressure on “amount of petroleum used”.) I also expect that there has been a discontinuity in “use of neural nets for machine learning”, and similarly I expect AI coding assistants will become hugely more popular in the nearish future. The relevant question to me is whether we saw a discontinuity in something like “ability to heat your home” or “ability to travel long distances” or something like that.
Printing: Going off of AI Impacts’ investigation, I’d count this one. I think partly this was because there wasn’t much effort going into this. (It looks like when the printing press was invented we were producing ~50,000 manuscripts per year, using about 25,000 person-years of labor. Presumably much much less than that was going into optimizing the process, similarly to how R&D in machine translation is way way lower than the size of the translation market.)
Spaceflight: Agree that this is a zero-to-one discontinuity from “physics hack” (but note that it did involve a huge amount of effort). Although if you’re saying that more resources were spent on it, same comment as petroleum.
Rockets: I’d love to see numbers, but this does sound like a discontinuity that’s relevant to the case with AI. I’d also want to know how much people cared about it (plausibly quite a lot).
Aluminium: Looking at AI Impacts, I think I’m at “probably a discontinuity relevant to the case with AI, but not a certainty”.
Radar: I’d need more details about what happened here, but it seems like this is totally consistent with the “continuous view” (since “with more effort you got more progress” seems like a pretty central conclusion of the model ).
Radio: Looking at AI Impacts, I think this one looks more like “lots of crazy fast progress that is fueled by frequent innovations”, which seems pretty compatible with the “continuous view” on AI. (Though I’m sympathetic to the critique that the double exponential curve chosen by AI Impacts is an instance of finding a line by which things look smooth; I definitely wouldn’t have chosen that functional form in advance of seeing the data.)
Automobile: I’d assume that there was very little optimization on “speed of production of cars” at the time, given that cars had only just become commercially viable, so a discontinuity seems unsurprising.
Transistors: Wikipedia claims “the MOSFET was also initially slower and less reliable than the BJT”, and further discussion seems to suggest that its benefits were captured with further work and effort (e.g. it was a twentieth the size of a BJT by the 1990s, decades after invention). This sounds like it wasn’t a discontinuity to me. What metric did you think it was a discontinuity for?
PCR: I don’t know enough about the field—sounds like a zero-to-one discontinuity (or something very close, where ~no one was previously trying to do the things PCR does). See aviation.
Conversation on technology forecasting and gradualism
Thanks, updated
[AN #170]: Analyzing the argument for risk from power-seeking AI
She needs an alien theory of mind to understand what the alien wants
Absolutely, I would think that the first order of business would be to learn that alien theory of mind (and be very conservative until that’s done).
Maybe you’re saying that this alien theory of mind is unlearnable, even for a very intelligent Alice? That seems pretty surprising, and I don’t feel the force of that intuition (despite the Occam’s razor impossibility result).
Notice that all those desiderata are much easier when the AI knows our (extrapolated) preferences. It is not clear at all that they can be achieved otherwise.
It seems like, as long as she wanted to, a human Alice could satisfy these desiderata when helping Bob, even though Alice doesn’t know Bob’s extrapolated preferences? So I’m not sure why you think an intelligent AI couldn’t do the same.
Maybe you think that it’s because Alice and Bob are both humans? But I also think Alice could satisfy these desiderata when helping an alien from a different planet—she would definitely make some mistakes, but presumably not the existentially catastrophic variety*.
*unless the alien has some really unusual values where an existential catastrophe can be caused by accident, e.g. “if anyone ever utters the word $WORD, that is the worst possible universe”, but those sorts of values seem very structurally different than human values.
There’s going to be a similar assumption in a good multipolar trap approach.
If Alice yelled at you with a megaphone that everyone in your house / city / whatever must now (1) obey Alice, and (2) torture anyone who doesn’t obey Alice, that’s not going to cause you to start obeying Alice.
You need some feedback signal that actually causes the agents to care about the rules; in debate this comes from the human judge + gradient descent, I expect you’ll need something analogous in any multipolar trap approach.