Communications lead at MIRI. Unless otherwise indicated, my posts and comments here reflect my own views, and not necessarily my employer’s.
Rob Bensinger
Karma: 18,148
A challenge for AGI organizations, and a challenge for readers
many approaches to training AGI currently seem to have as a training target something like “learn to predict humans”, or some other objective that is humanly-meaningful but not-our-real-values,
I don’t know whether this will continue in the future (all the way up to AGI). If it does, then it strikes me as a sufficiently coarse-grained approach (that’s bad enough at inner alignment, and bad enough at outer-alignment-to-specific-things-we-actually-care-about) that I’d still be pretty surprised if the result (in the limit of superintelligence) bears any resemblance to stuff we care much about, good or bad.
E.g., there are many more “unconscious configurations of matter that bear some relation to things you learn in trying to predict humans” than there are “conscious configurations of matter that bear some relation to things you learn in trying to predict humans”. Building an entire functioning conscious mind is still a very complicated end-state that requires getting lots of bits into the AGI’s terminal goals correctly; it doesn’t necessarily become that much easier just because we’re calling the ability we’re training “human prediction”. Like, a superintelligent paperclipper would also be excellent at the human prediction task, given access to information about humans.
(I’ll also mention that I think it’s a terrible idea for safety-conscious AI researchers to put all their eggs in the “train AI via lots of data on humans” basket. But that’s a separate question from what AI researchers are likely to do in practice.)
How are you defining “super-intelligent”, “AGI”, and “ANI” here?
I’d distinguish two questions:
Pivotal-act-style AI: How do we leverage AI (or some other tech) to end the period where humans can imminently destroy the world with AI?
CEV-style AI: How do we leverage AI to solve all of our problems and put us on a trajectory to an ~optimal future?
My guess is that successful pivotal act AI will need to be AGI, though I’m not highly confident of this. By “AGI” I mean “something that’s doing qualitatively the right kind of reasoning to be able to efficiently model physical processes in general, both high-level and low-level”.
I don’t mean that the AGI that saves the world necessarily actually has the knowledge or inclination to productively reason about arbitrary topics—e.g., we might want to limit AGI to just reasoning about low-level physics (in ways that help us build tech to save the world), and keep the AGI from doing dangerous things like “reasoning about its operators’ minds”. (Similarly, I would call a human a “general intelligence” insofar as they have the right cognitive machinery to do science in general, even if they’ve never actually thought about physics or learned any physics facts.)
In the case of CEV-style AI, I’m much more confident that it will need to be AGI, and I strongly expect it to need to be aligned enough (and capable enough) that we can trust it to reason about arbitrary domains. If it can safely do CEV at all, then we shouldn’t need to restrict it—needing to restrict it is a flag that we aren’t ready to hand it such a difficult task.
The definitions given in the post are:
ASI-boosted humans — We solve all of the problems involved in aiming artificial superintelligence at the things we’d ideally want.
[...]
misaligned AI — Humans build and deploy superintelligent AI that isn’t aligned with what we’d ideally want.
I’d expect most people to agree that “We solve all of the problems involved in aiming artificial superintelligence at the things we’d ideally want” yields outcomes that are about as good as possible, and I’d expect most of the disagreement to turn (either overtly or in some subtle way) on differences in how we’re defining relevant words (like “ideally”, “good”, and “problems”).
I’d be fine with skipping over this question, except that some of the differences-in-definition might be important for the other questions, so this question may be useful for establishing a baseline.
With “misaligned AI”, there are some definitional issues but I expect most of the disagreement to be substantive, since there are a lot of different levels of Badness you could expect even if you want to call all misaligned AI “bad” (at least relative to ASI-boosted humans).
In my own answers, I interpreted “misaligned AGI” as meaning: We weren’t good enough at alignment to make the AGI do exactly what we wanted, so it permanently took control of the future and did “something that isn’t exactly what we wanted” instead. (Which might be kinda similar to what we wanted, or might be wildly different, etc.)
If an alien only cared about maximizing the amount of computronium in the universe, and it built an AI that fills the universe with computronium because the AI values calculating pi, then I think I’d say that the AI is “aligned with that alien by default / by accident”, rather than saying “the AI is misaligned with that alien but is doing ~exactly what we want anyway”. So if someone thinks AI does exactly what humans want even with humans putting in zero effort to steer the AI toward that outcome, I’d classify that as “aligned-by-default AI”, rather than “misaligned AI”. (But there’s still a huge range of possible-in-principle outcomes from misaligned AI, even if I think some a lot more likely than others.)
From the scaling-pilled perspective, or even just centrist AI perspective, this is an insane position: it is taking a L on one of, if not the most, important future technological capabilities, which in the long run may win or lose wars. If China wants to dominate Asia, much less surpass the obsolete American empire, or create AGI, or lead in aerospace, or create ‘5G’ or whatever, it’s hard to see how it’s going to do that while paying more for chips which are half a decade or worse out of date.
The scaling-pilled AI view ought to be that scaling AI kills you. Why pretend that there’s a strategic advantage here, as opposed to a loaded gun you can point at your own head if you’re stupid enough?
It’s one thing to say “given China’s actual beliefs, they ought to do X” or “if China were rationally acting on a correct understanding of the world, they would do Y”. But why criticize China for avoiding a self-destructive action that would make sense to do if they had a specific combination of definitely-true, maybe-true, and definitely-false beliefs—a specific combination they don’t in fact have?
Sounds right to me! I dunno Nate’s view on this.
That’s what he’s claiming, because he’s claiming “cosmopolitan value” is itself a human value. (Just a very diversity-embracing one.)
I think Dennett’s argumentation about the hard problem of consciousness has usually been terrible, and I don’t see him as an important forerunner of illusionism, though he’s an example of someone who soldiered on for anti-realism about phenomenal consciousness for long stretches of time where the arguments were lacking.
I think I remember Eliezer saying somewhere that he also wasn’t impressed with Dennett’s takes on the hard problem, but I forget where?
His approach also seems to be “explain our claims about consciousness”.
There’s some similarity between heterophenomenology and the way Eliezer/Nate talk about consciousness, though I guess I think of Eliezer/Nate’s “let’s find a theory that makes sense of our claims about consciousness” as more “here’s a necessary feature of any account of consciousness, and a plausibly fruitful way to get insight into a lot of what’s going on”, not as an argument for otherwise ignoring all introspective data. Heterophenomenology IMO was always a somewhat silly and confused idea, because it’s proposing that we a priori reject introspective evidence but it’s not giving a clear argument for why.
(Or, worse, it’s arguing something orthogonal to whether we should care about introspective evidence, while winking and nudging that there’s something vaguely unrespectable about the introspective-evidence question.)
There are good arguments for being skeptical of introspection here, but “that doesn’t sound like it’s in the literary genre of science” should not be an argument that Bayesians find very compelling.
Thanks for the comment, Amelia! :)
One million years is ten thousand generations of humans as we know them. If AI progress were impossible under the heel of a world-state, we could increase intelligence by a few points each generation. This already happens naturally and it would hardly be difficult to compound the Flynn effect.
I think the “unboosted humans” hypothetical is meant to include mind-uploading (which makes the generation time an underestimate), but we’re assuming that the simulation overlords stop us from drastically improving the quality of our individual reasoning.
Nate assigns “base humans, left alone” an ~82% chance of producing an outcome at least as good as “tiling our universe-shard with computronium that we use to run glorious merely-human civilizations”, which seems unlikely to me if we can’t upload humans at all. (But maybe I’m misunderstanding something about his view.)
Surely we could hit endgame technology that hits the limits of physical possibility/diminishing returns in one million years, let alone five hundred of those spans.
I think we hit the limits of technology we can think about, understand, manipulate, and build vastly earlier than that (especially if we have fast-running human uploads). But I think this limit is a lot lower than the technologies you could invent if your brain were as large as the planet Jupiter, you had native brain hardware for doing different forms of advanced math in your head, you could visualize the connection between millions of different complex machines in your working memory and simulate millions of possible complex connections between those machines inside your own head, etc.
Even when it comes to just winning a space battle using a fixed pool of fighters, I expect to get crushed by a superintelligence that can individually think about and maneuver effectively arbitrary numbers of nanobots in real time, versus humans that are manually piloting (or using crappy AI to pilot) our drones.
In comparative terms, a five hundred year sabbatical from AI would reduce the share of resources we could reach by an epsilon only, and if AI safety premises are sound then it would greatly increase EV.
Oh, agreed. But we’re discussing a scenario where we never build ASI, not one where we delay 500 years.
This point is likely moot, of course. I understand that we do not live in a totalitarian world state and your intent is just to assure people that AI safety people are not neoluddites.
Yep! And more generally, to share enough background model (that doesn’t normally come up in inside-baseball AI discussions) to help people identify cruxes of disagreement.
I suppose one could attempt to help a state establish global dominance
Seems super unrealistic to me, and probably bad if you could achieve it.
A different scenario that makes a lot more sense, IMO, is an AGI project pairing with some number of states during or after an AGI-enabled pivotal act. But that assumes you’ve already solved enough of the alignment problem to do at least one (possibly state-assisted) pivotal act.
I think there’s kind of a lot of room between 95% of potential value being lost and 5%!!
My intuition is that capturing even 1% of the future’s total value is an astoundingly conjunctive feat—a narrow enough target that it’s surprising if we can hit that target and yet not hit 10%, or 99%. Think less “capture at least 1% of the negentropy in our future light cone and use it for something”, more “solve the first 999 digits of a 1000-digit decimal combination lock specifying an extremely complicated function of human brain-states that somehow encodes all the properties of Maximum Extremely-Weird-Posthuman Utility”.
(This is based on the idea that even if the alignment problem is solved such that we know how to specify a goal rigorously to an AI, it doesn’t follow that the people who happen to be programming the goal will be selfless. You work in AI so presumably you have practiced rebuttals to this concept; I do not so I’ll state my thought but be clear that I expect this is well-worn territory to which I expect you to have a solid answer.)
Why do they need to be selfless? What are the selfish benefits to making the future less Fun for innumerable numbers of posthumans you’ll never meet or hear anything about?
(The future light cone is big, and no one human can interact with very much of it. You swamp the selfish desires of every currently-living human before you’ve even used up the negentropy in one hundredth of a single galaxy. And then what do you do with the rest of the universe? We aren’t guaranteed to use the rest of the universe well, but if we use it poorly the explanation probably can’t be “selfishness”.)
It seems to assume that things like hive-mind species are possible or common, which I don’t have information about but maybe you do.
I dunno Nate’s reasoning, but AFAIK the hive-mind thing may just be an example, rather than being central to his reasoning on this point.
Luke M and I are illusionists, but I don’t think Eliezer or Nate are illusionists.
Oooh, sounds right to me!
Sounds right to me!
Predictions, using the definitions in Nate’s post:
Strong Utopia Will we achieve strong utopia given ASI-boosted humans? Will we achieve strong utopia given unboosted humans? Will we achieve strong utopia given ASI-boosted aliens? Will we achieve strong utopia given misaligned AI? Weak Utopia Will we achieve weak utopia given ASI-boosted humans? Will we achieve weak utopia given unboosted humans? Will we achieve weak utopia given ASI-boosted aliens? Will we achieve weak utopia given misaligned AI? “Pretty good” outcome Will we achieve a "pretty good" outcome given ASI-boosted humans? Will we achieve a "pretty good" outcome given unboosted humans? Will we achieve a "pretty good" outcome given ASI-boosted aliens? Will we achieve a "pretty good" outcome given misaligned AI? Conscious Meh outcome Will we achieve a "conscious meh" outcome given ASI-boosted humans? Will we achieve a "conscious meh" outcome given unboosted humans? Will we achieve a "conscious meh" outcome given ASI-boosted aliens? Will we achieve a "conscious meh" outcome given misaligned AI? Unconscious Meh outcome Will we achieve a "unconscious meh" outcome given ASI-boosted humans? Will we achieve a "unconscious meh" outcome given unboosted humans? Will we achieve a "unconscious meh" outcome given ASI-boosted aliens? Will we achieve a "unconscious meh" outcome given misaligned AI? Weak dystopia Will we end up with a weak dystopia given ASI-boosted humans? Will we end up with a weak dystopia given unboosted humans? Will we end up with a weak dystopia given ASI-boosted aliens? Will we end up with a weak dystopia given misaligned AI? Strong dystopia Will we end up with a strong dystopia given ASI-boosted humans? Will we end up with a strong dystopia given unboosted humans? Will we end up with a strong dystopia given ASI-boosted aliens? Will we end up with a strong dystopia given misaligned AI?
To me it seems like destroying a bunch of sentient minds and then recreating an equivalent number later
Destroying, minus all of the experiential bad things we mentally associate with “destroying”!
I mean, why wouldn’t it? Weird differences at that low a level having profound differences on something high-level like utility functions feels normal to me?
I don’t object to the idea of people having utility functions over various unconscious physical states—e.g., maybe I assign some aesthetic value to having my upload run on cool shiny-looking computronium, rather than something tacky, even if no one’s ever going to see my hardware.
The three things that make me suspicious of your “this is a huge part of my utility function” claim are:
The fact that the properties you’re attached to are things that (for all you know) you never had in the first place. The idea of humans experiencing and enjoying beauty, and then desiring that things be beautiful even in observed parts of the universe, makes total sense to me. The idea of humans coming up with a weird metaphysical hypothesis, and assigning ultimate value to the answer to this hypothesis even though its truth or falsehood has never once made a perceptible difference in anyone’s life, seems far weirder to me.
This alleged utility function component happens to coincide with a powerful and widespread cognitive illusion (the idea of a Soul or Self or Homunculus existing outside determinism / physics / the Cartesian theater).
The alleged importance of this component raises major alarm bells. I wouldn’t be skeptical if you said that you slightly aesthetically prefer bodily continuity, all else equal; but treating this metaphysical stuff as though it were a matter of life or death, or as though it were crucial in the way that the distinction between “me” and “a random stranger” is important, seems like a flag that there’s something wrong here. The “it’s a matter of life or death” thing only makes sense, AFAICT, if you’re subject to a soul-style illusion about what “you” are, and about what your experience of personal continuity currently consists in.
You see this person, you interact with him (albeit in weird, perfectly symmetrical ways), would you have no preference for your life and happiness over his?
To my eye, this is isomorphic to the following hypothetical (modulo the fact that two copies of me existing may not have the same moral value as one copy of me existing):
I’m playing a VR game.
Iff a VR avatar of mine is killed, I die in real life.
When I use the controls, it causes two different avatars to move. They move in exactly the same way, and since they’re symmetrically positioned in a symmetrical room, there’s no difference between whether I’m seeing through the eyes of one avatar or the other. Through my VR helmet, I am in fact looking through the eyes of just one of the two avatars—I can’t see both of their experiences at once—but since the experiences are identical, it doesn’t matter which one.
It doesn’t really matter whether I value “my” life over “his”, because we’re going to have the same experiences going forward regardless. But if something breaks this symmetry (note that at that point we’d need to clarify that there are two separate streams of consciousness here, not just one), my preference before the symmetry is broken should be indifferent between which of the copies gets the reward, because both versions of me bear the correct memory-relationship to my present self.
(So, I should “anticipate” being both copies of me, for the same reason I should “anticipate” being all of my MWI selves, and anticipate having new experiences after stepping through a teleporter.)
with a caveat that it’s possible this happens in a way that preserves the stream-of-consciousness I care about in some way
??? You say that this is a claim you feel uncertain about. Does this uncertainty cash out in any experiential difference?
Like, the mere fact that we don’t know whether we’re being destroyed and recreated innumerable times every second, seems to rule out the idea that there’s an experiential difference here.
We could think of ourselves as characters in a movie, whose state at each moment is represented as a frame in a film reel. From within the movie, no character can perceive the transition from one frame to the next, precisely because the brain-state encoded in each frame is our experience at that moment. We experience the frames, not the transitions or the superstructure those frames are part of. Expecting a denizen of a film frame to “notice” whether (e.g.) you double up on every frame, is like expecting them to “notice” that there’s a rubber duck a few feet away from the film reel, outside of the movie universe.
Given that there’s a smallest “physically meaningful” unit of time (the Planck time), it’s not clear to me that it makes sense to think of your physical body as continuously persisting over time, as opposed to “jumping” (film-reel-style, or flipbook-style) from frame to frame. But maybe you think that’s fine, because a body that persists via jerky, uncontinuous motions (at some level of granularity) is still different from one that’s annihilated and recreated in between each jerk?
I assume you also, then, think it would be horrible news to think you’re in a simulation? Since by the same logic as “it would be tragic to learn that my body is constantly being destroyed and recreated” (even though this would, by assumption, be how things had always been, would be normality and inevitable, and wouldn’t change a danged thing about what you experience at any time), it would surely also be tragic to learn that your body never existed (even though the same “this makes no experiential difference” caveats hold)?
To my eye, all of this smacks of a soul-ish illusion. I get the intuition that, e.g., it’s weird to “anticipate” the future experiences of a version of me that went through a teleporter (being destroyed, and then recreated somewhere else). But our memories are exactly the thing that creates the impression of a continuous self—as noted above, there would be literally no experiential difference if we learned that our body were a simulation, or that it were being constantly destroyed and recreated. It’s just the accumulation of memory that makes any of this feel, at any given time, “continuity-of-self-ish” / “stream-of-consciousness-ish”.
And the teletransporter would preserve memory, to exactly the same degree that any routine action I take preserves memory. The idea that there’s a ghost in my brain that won’t get to really experience the post-teleportation experience, even though it feels obvious that this ghost is surviving moment-to-moment as I wrote this LessWrong comment, has to be an illusion, because there is no Bayesian evidence I have that my current moment-to-moment experience is any different from the teletransporter case. My brain has not encountered a single experience, in its entire lifetime, that differentiates the “moment-to-moment existence involves Me persisting over time” hypothesis from the “moment-to-moment existence involves a succession of Me-like entities replacing each other while passing on the memories of the previous Me”.
The “succession of Me entities persisting over time, with memories preserved” hypothesis is completely empirically indistinguishable from the “one Me persisting over time, with memories preserved” hypothesis. So it seems clear to me that I should “anticipate” having an experience a moment from now iff it has the same kind of memory-relation to my present experience. It will feel the same in any case, if you take a survey of all the experiences happening in the universe at a given time. The idea that there’s a Soul, separate from my experiences and memories, that will blip out of existence or perceive Darkness if something interferes with its Adhesion to a body (even though the exact same memory-relation to a body in the world still holds) -- this idea doesn’t make physical sense.
And since it’s just a free-floating intuition, not a thing that we could possibly have gotten Bayesian evidence for, it seems unusually clear that it has to be an illusion.
By the same logic, if more than one “me” is created with that same memory-relation to me, then I should “anticipate” having all of those experiences. Not in the sense that I’m a Soul that will get to stare at multiple movie screens simultaneously; but in the sense that the experiencing self will itself be copied, and each movie screen will get its own “me” that has an equal claim to being “the same person” as present-me. See also the MWI example.
The experiencing self is in the universe—in the film reel, in the momentary brain-state—and this feels emotionally un-obvious (and the teletransporter feels scary) because of an introspective illusion. If this is true, then it seems unparsimonious to separately claim that our CEV would assign enormous moral importance to this weird metaphysical thingie we currently have an illusion about; we would expect the illusion to feel emotionally salient even if our CEV doesn’t care about bodily continuity.
My example with the 100 million referred to question 1.
Yeah, I’m also talking about question 1.
I do think that stuff only matters (to me) if it’s in some sense causally connected to my life and experiences.
Seems obviously false as a description of my values (and, I’d guess, just about every human’s).
Consider the simple example of a universe that consists of two planets: mine, and another person’s. We don’t have spaceships, so we can’t interact. I am not therefore indifferent to whether the other person is being horribly tortured for thousands of years.
If I spontaneously consider the hypothetical, I will very strongly prefer that my neighbor not be tortured. If we add the claims that I can’t affect it and can’t ever know about it, I don’t suddenly go “Oh, never mind, fuck that guy”. Stuff that happens to other people is real, even if I don’t interact with it.
This is why I suggested naming FDT “functional decision theory” rather than “algorithmic decision theory”, when MIRI was discussing names.
Suppose that Alice is an LDT Agent and Bob is an Alien Agent. The two swap source code. If Alice can verify that Bob (on the input “Alice’s source code”) behaves the same as Alice in the PD, then Alice will cooperate. This is because Alice sees that the two possibilities are (C,C) and (D,D), and the former has higher utility.
The same holds if Alice is confident in Bob’s relevant conditional behavior for some other reason, but can’t literally view Bob’s source code. Alice evaluates counterfactuals based on “how would Bob behave if I do X? what about if I do Y?”, since those are the differences that can affect utility; knowing the details of Bob’s algorithm doesn’t matter if those details are screened off by Bob’s functional behavior.
Depends on the decision algorithm! Do you have a specific one in mind?
E.g., LDT will defect against CDT.
it isn’t me in the stream-of-consciousness-that-forms-my-self sense
How do you know?
If it turned out that the entire universe were constantly being destroyed and then an exact copy recreated, innumerable times every second, would you thereby learn something crucial about the value of copies of yourself? Why would a random weird metaphysics footnote like this make such a profound difference for your utility function?
Seems even weirder to me given how much turnover there is in a body’s composition over time. Caring about your future self as long as you slowly switch out your body parts for functionally identical new parts, but not if you do the same process quickly, feels to me like deciding that getting a haircut destroys your continuity of self. (But weirder, because at least a haircut makes a perceptible difference of some sort!)
Do you think that there are specific falsehoods in the OP? Or do you just think it’s unrespectable for humans to think about the future?
Some people object to working on AGI alignment on the grounds that the future will go a lot better if we take our hands off the steering wheel and let minds develop “naturally” and “freely”.
Some of those people even own companies with names like “Google”!
The best way to address that family of views is to actually talk about what would probably happen if you let a random misaligned AGI, or a random alien, optimize the future.
So on your understanding, “foxes” = people who have One Big Theory about which topics are respectable, and answer all futurist questions based on that theory? While “hedgehogs” = people who write long, detailed blog posts poking at various nuances and sub-nuances of a long list of loosely related object-level questions?
… Seems to me that you’re either very confused about what “foxes” and “hedgehogs” are, or you didn’t understand much of the OP’s post.
Writing a long post about a topic doesn’t imply that you’re using One Simple Catch-All Model to generate all the predictions, and it doesn’t imply that you’re confident about the contents of the post. Refusing to think about a topic isn’t being a “fox”.
Because as all foxes know, “thinking about the present” and “thinking about the future” are mutually exclusive.