Alex Turner, postdoctoral researcher at the Center for Human-Compatible AI. Reach me at turner.alex[at]berkeley[dot]edu.
TurnTrout
Karma: 9,975
Argument that you can’t use a boundedly intelligent ELK solution to search over plans to find one which keeps the diamond in the vault. That is, the ELK solution probably would have to be at least as smart (or smarter) than the plan-generator.
Consider any situation where it’s hard to keep the diamond in the vault. Then any successful plan will have relatively few degrees of freedom. Like, a bunch of really smart thieves will execute a cunning plot to extract the diamond. You can’t just sit by or deploy some simple traps in this situation.
Therefore, any plan which keeps the diamond in the vault, will be “smart” in some way.
Now notice that it’s possible to build a successor AI which happens to trick the oracle (i.e. the assumed ELK solution) into believing the diamond is still in the vault. The reason is that even though the oracle is superintelligent, it is still flawed in unforeseeable-to-us ways, its mind has “cracks” which can be exploited by unpredictable-to-us adversarial inputs (loose analogy: epilepsy). Inputs which would be findable by brute-force-search. So there exist plans which trick the oracle. Even if the oracle understands the dangers of successor AIs in general, even if the oracle is cautious in general, any powerful enough brute force search will expose exploits in its cognition.
Now, consider that if object-level diamond-protecting plans have to be sufficiently complex in order to retain the diamond, they might be far longer than plans which simply build a successor AI which takes over the world and tricks the oracle into saying “the diamond is still in the room”. Such AI-creation plans would have many degrees of freedom, since such AIs probably have low K-complexity and admit short programs (findable by brute force).
Therefore, if you just do brute-force search over all plans, and randomly sample one where the oracle returns
yes the diamond is still there, there would be exponentially many more plans which have a successor AI, than which involve protecting the diamond for real. And so somehow we still die to an unaligned AI, even given the ELK oracle, and the diamond probably doesn’t even stay in the room.I was prompted into realizing this by some mixture of talking with Rohin Shah and John Wentworth. Maybe most of the idea was John’s (though he might not agree with my recounting of our discussion).
But anyways, in the moment of realization, my suspension of disbelief got unsuspended. I snapped. I thought “there is actually no way that any of this is on the real path[1] to solving alignment, this is just too insane.” (And I began looking for other frames on the alignment problem, preparing to rethink the problem from scratch after I graduated.)
- ^
Brute-force-search probably isn’t what ELK proponents would suggest. But it sure seems like, given an ELK solution, you should be able to simply find plans which at least keep the diamond in the room, without a whole bunch more alignment work.
- ^
Before reading, here are my reactions to the main claims:
Not only does RL not, by default, produce policies which have reward maximization as their behavioral objective, but in fact I argue that it is not possible for RL policies to care about “reward” in an embedded setting. (agreed under certain definitions of “reward”)
I argue that this does not imply that wireheading in RL agents is impossible, because wireheading does not mean “the policy has reward as its objective”.It is still possible for an RL agent to wirehead, and in fact, is a high probability outcome under certain circumstances. (unsure, depends on previous definition)
This bears a clean analogy to the human case, viewing humans as RL agents; there is no special mechanism going on in humans that is needed to explain why humans care about things in the world. (agreed insofar as I understand the claim)
(I ended up agreeing with these points by the end of the post!)
There even exists a policy that cares about that particular register at that particular physical location in the world even if it is not connected to the RL algorithm at all, but that policy does not get selected for by the RL algorithm.
Why not? Selection only takes place during training, the RL algorithm can only select across policies while the RL algorithm is running. A policy which wireheads and then disconnects the RL algorithm is selection-equivalent to a policy which wireheads and then doesn’t disconnect the RL algorithm (mod the cognitive-updates provided by reward in the latter case, but that seems orthogonal).
This resolves the question of why humans don’t want to wirehead—because you identify with the rest of the brain and so “your” desires are the mesaobjectives.
Agreed!
In particular, one consequence of this is we also don’t need to postulate the existence of some kind of special as yet unknown algorithm that only exists in humans to be able to explain why humans end up caring about things in the world. Whether humans wirehead is determined by the same thing that determines whether RL agents wirehead.
Agreed. Shard theory isn’t postulating such an algorithm, if that’s what you meant to imply. In fact, your entire end of the post seems to be the historical reasoning which Quintin and I went through in developing shard theory 101.
I’m often asked, Why “shard theory”? I suggested this name to Quintin when realizing that human values have the type signature of contextually activated decision-making influences. The obvious choice, then, was to call these things “shards of value”—drawing inspiration from Eliezer’s Thou art godshatter, where he originally wrote “shard of desire.”
(Contrary to several jokes, the choice was not just “because ‘shard theory’ sounds sick.”)
This name has several advantages. Value-shards can have many subshards/facets which vary contextually (a real crystal may look slightly different along its faces or have an uneven growth pattern); value-shards grow in influence over time under repeated positive (just as real crystals can grow); value-shards imply a degree of rigidity, but also incompleteness—they are pieces of a whole (on my current guess, the eventual utility function which is the reflective equilibrium of value-handshakes between the set of endorsed shards which bid as a function of their own future prospects). Lastly, a set of initial shards will generally steer the future towards growing themselves (e.g. banana-shard leads to more banana-consumption → more reward → the shard grows and becomes more sophisticated); similarly, given an initial start of a crystalline lattice which is growing, I’d imagine it becomes more possible to predict the later lattice configuration due to the nature of crystals.
I think that shards will cast contextual shadows into the factors of a person’s equilibrated utility function, because I think the shards are contextually activated to begin with. For example, if a person hates doing jumping jacks in front of a group of her peers, then that part of herself can bargain to penalize jumping jacks just in those contexts in the final utility function. Compared to a blanket “no jumping jacks ever” rule, this trade is less costly to other shards and allows more efficient trades to occur.
A dangerous intuition pump here would be something like, “If you take a human who was trained really hard in childhood to have faith in God and show epistemic deference to the Bible, and inspecting the internal contents of their thought at age 20 showed that they still had great faith, if you kept amping up that human’s intelligence their epistemology would at some point explode”
Yes, a value grounded in a factual error will get blown up by better epistemics, just as “be uncertain about the human’s goals” will get blown up by your beliefs getting their entropy deflated to zero by the good ole process we call “learning about reality.” But insofar as corrigibility is “chill out and just do some good stuff without contorting 4D spacetime into the perfect shape or whatever”, there are versions of that which don’t automatically get blown up by reality when you get smarter. As far as I can tell, some humans are living embodiments of the latter. I have some “benevolent libertarian” values pushing me Pareto improving everyone’s resource counts and letting them do as they will with their compute budgets. What’s supposed to blow that one up?
that in real life if we were faced with a very bizarre alien we would be unlikely to want to defer to it. Our lack of scalable desire to defer in all ways to an extremely bizarre alien that ate babies, is not something that you could fix just by giving us an emotion of great deference or respect toward that very bizarre alien. We would have our own thought processes that were unlike its thought processes, and if we scaled up our intelligence and reflection to further see the consequences implied by our own thought processes, they wouldn’t imply deference to the alien even if we had great respect toward it and had been trained hard in childhood to act corrigibly towards it.
This paragraph as a whole seems to make a lot of unsupported-to-me claims and seemingly equivocates between the two bolded claims, which are quite different. The first is that we (as adult humans with relatively well-entrenched values) would not want to defer to a strange alien. I agree.
The second is that we wouldn’t want to defer “even if we had great respect toward it and had been trained hard in childhood to act corrigibly towards it.” I don’t see why you believe that. Perhaps if we were otherwise socialized normally, we would end up unendorsing that value and not deferring? But I conjecture if that a person weren’t raised with normal cultural influences, you could probably brainwash them into being aligned baby-eaters via reward shaping via brain stimulation reward.
Acting corrigibly towards a baby-eating virtue ethicist when you are a utilitarian is an equally weird shape for a decision theory.
A utilitarian? Like, as Thomas Kwa asked, what are the type signatures of the utility functions you’re imagining the AI to have? Your comment makes more sense to me if I imagine the utility function is computed over “conventional” objects-of-value.
You seem to mostly be imagining a third category:
3. If you optimize a model to be corrigible in one situation, how likely is it to still be corrigible in a new situation?
I don’t care about question 3. It’s been more than 4 years since I even seriously discussed the possibility of learning on a mechanism like that, and even at that point it was not a very serious discussion.
“Don’t care” is quite strong. If you still hold this view—why don’t you care about 3? (Curious to hear from other people who basically don’t care about 3, either.)
I agree that similar environments are important, but I don’t see why you think they explain most of the outcomes. What’s an example of a “gene-environment interaction with parts of the environment that aren’t good at predicting the child’s moral development”?
Like, what it feels like to understand human inductive biases isn’t to think “Gee, I understand inductive biases!”. It’s more like: “I see that my son just scowled after agreeing to clean his room. This provides evidence about his internal motivational composition, even though I can’t do interpretability and read off his brain state. Given what I know about human physiology and people like him, I predict that while he might comply now (‘in training’), at which point I will reward him with snacks. However, in the future he will probably not generalize to actually cleaning his room when asked when I’m less able to impose external punishments and rewards.”
I also claim that a human would be relatively easy to reshape into caring about baby-eaters if you used a competent and unethical (psychological) shaping scheme and hidden brain stimulation reward devices, such that you super strongly reinforced the human when they indicate they just thought positive or charitable thoughts about baby eaters, or agree that the baby eaters deserve moral consideration. I think you could probably pull this off within a week.
Now, we haven’t actually observed this. But insofar as you agree with my prediction, it’s not due to an environment-gene interaction. This is what it feels like to understand inductive biases: Being able to correctly predict how to inculcate target values in another agent, without already having done it experimentally.
Sometimes in software development, you may be worried that there is a security problem in the program you are making. But if you speak out loud about it, then that generates FUD among the users, which discourages you from speaking loud in the future. Hence, RLHF in a human context generates deception.
Thanks for the example. The conclusion is far too broad and confident, in my opinion. I would instead say “RLHF in a human context seems to have at least one factor which pushes for deception in this kind of situation.” And then, of course, we should compare the predicted alignment concerns in people, with the observed alignment situation, and update accordingly. I’ve updated down hard on alignment difficulty when I’ve run this exercise in the past.
Not necessarily a general deception shard, just it spawns some sort of shard that repeats similar things to what it did before, which presumably means more often errorneously making us believe it saved a person’s life.
I don’t see why this is worth granting the connotations we usually associate with “deception”
I think that if the AI just repeats “I saved someone’s life” without that being true, we will find out and stop rewarding that?
Unless the AI didn’t just happen to get erroneously rewarded for prosociality (as originally discussed), but planned for that to happen, in which case it’s already deceptive in a much worse way.
But somehow the AI has to get to that cognitive state, first. I think it’s definitely possible, but not at all clearly the obvious outcome.
Your post points out that you can do all sorts of things in theory if you “have enough write access to fool credit assignment”. But that’s not sufficient to show that they can happen in practice. You gotta propose system of write access and training to use this write access to do what you are proposing.
That wasn’t the part of the post I meant to point to. I was saying that just because we externally observe something we would call “deception/misleading task completion” (e.g. getting us to reward the AI for prosociality), does not mean that “deceptive thought patterns” get reinforced into the agent! The map is not the territory of the AI’s updating process. The reward will, I think, reinforce and generalize the AI’s existing cognitive subroutines which produced the judged-prosocial behavior, which subroutines don’t necessarily have anything to do with explicit deception (as you noted).
Would you not agree that models are unaligned by default, unless there is something that aligns them?
Is a donut “unaligned by default”? The networks start out randomly initialized. I agree that effort has to be put in to make the AI care about human-good outcomes in particular, as opposed to caring about ~nothing, or caring about some other random set of reward correlates. But I’m not assuming the model starts out deceptive, nor that it will become that with high probability. That’s one question I’m trying to figure out with fresh eyes.
Shard theory suggests that goals are more natural to specify/inculcate in their shard-forms (e.g. if around trash and a trash can, put the trash away), and not in their (presumably) final form of globally activated optimization of a coherent utility function which is the reflective equilibrium of inter-shard value-handshakes (e.g. a utility function over the agent’s internal plan-ontology such that, when optimized directly, leads to trash getting put away, among other utility-level reflections of initial shards).
I could (and did) hope that I could specify a utility function which is safe to maximize because it penalizes power-seeking. I may as well have hoped to jump off of a building and float to the ground. On my model, that’s just not how goals work in intelligent minds. If we’ve had anything at all beaten into our heads by our alignment thought experiments, it’s that goals are hard to specify in their final form of utility functions.
I think it’s time to think in a different specification language.
I am a time-traveler who came back from the world where it (super duper predictably) turned out that a lot of early bright hopes didn’t pan out and various things went WRONG and alignment was HARD and it was NOT SOLVED IN ONE SUMMER BY TEN SMART RESEARCHERS
I think these kinds of comments update readers’ beliefs in a bad, invalid way. The bad event (AGI ruin) is argued for by… a request for me to condition on testimony of a survivor of that bad event. Yes, I know the whole thing is tongue-in-cheek. I know that EY is not literally claiming to be a time-traveller.
But in TurnTrout-culture, “I experienced X” is something to be said when X has actually been experienced. “The fact that X” is to be said when X is actually supported by a heap of accepted evidence.[1] “Have you met dath ilani?” is to be said when such entities actually exist and are not outputs of the model of intelligence which is being argued for. (Yes, that last one was flagged as a “bad argument”, but still.)
This paragraph of EY self-fic didn’t update me at all. But it almost did. When these statements are made, I am inclined to update my beliefs in the predictable way—to gullibly update on claims—unless I take special effort to not update on (checks dialogue) fictional evidence. Which effort I do take (as a matter of reflex, at this point), but that effort is a cost imposed on me.
- ^
This particular kind of misleading statement wasn’t made in this dialogue, but I’ve seen it made erroneously-according-to-me in private correspondence with smart researchers.
- ^
“Globally activated consequentialist reasoning is convergent as agents get smarter” is dealt an evidential blow by von Neumann:
Although von Neumann unfailingly dressed formally, he enjoyed throwing extravagant parties and driving hazardously (frequently while reading a book, and sometimes crashing into a tree or getting arrested). He once reported one of his many car accidents in this way: “I was proceeding down the road. The trees on the right were passing me in orderly fashion at 60 miles per hour. Suddenly one of them stepped in my path.” He was a profoundly committed hedonist who liked to eat and drink heavily (it was said that he knew how to count everything except calories). -- https://www.newworldencyclopedia.org/entry/John_von_Neumann
The most straightforward way to produce evidence of a model’s deception is to find a situation where it changes what it’s doing based on the presence or absence of oversight. If we can find a clear situation where the model’s behavior changes substantially based on the extent to which its behavior is being overseen, that is clear evidence that the model was only pretending to be aligned for the purpose of deceiving the overseer.
This isn’t clearly true to me, at least for one possible interpretation: “If the AI knows a human is present, the AI’s behavior changes in an alignment-relevant way.” I think there’s a substantial chance this isn’t what you had in mind, but I’ll just lay out the following because I think it’ll be useful anyways.
For example, I behave differently around my boss than alone at home, but I’m not pretending to be anything for the purpose of deceiving my boss. More generally, I think it’ll be quite common for different contexts to activate different AI values. Perhaps the AI learned that if the overseer is watching, then it should complete the intended task (like cleaning up a room), because historically those decisions were reinforced by the overseer. This implies certain bad generalization properties, but not purposeful deception.
You could say that sunflowers are very similar to other good things in the past, e.g. Beautiful Calm Nature and Movies and Flowers As Something Good and Places That Are 100% Not School or Workplace or Busy City. And I think that Shard Theory is actually in trouble either way:
How does that reinforcement event history create a sunflower-timidity-shard?
I haven’t fully understood all of your points, but they gloss as reasonable and good. Thank you for this high-effort, thoughtful comment!
(If anyone is interested in doing research on the evolution of prosocality vs antisocialness in humans and/or how these things might play out in AI training environments, I know people who would likely be interested in funding such work.)
I encourage applicants to also read Quintin’s Evolution is a bad analogy for AGI (which I wish more people had read, I think it’s quite important). I think that evolution-based analogies can easily go astray, for reasons pointed out in the essay. (It wasn’t obvious to me that you went astray in your comment, to be clear—more noting this for other readers.)
I was more wondering about situations in humans which you thought had the potential to be problematic, under the RLHF frame on alignment.
Like, you said:
shard theory resembles RLHF, and seems to share its flaws
So, if some alignment theory says “this approach (e.g. RLHF) is flawed and probably won’t produce human-compatible values”, and we notice “shard theory resembles RLHF”, then insofar as shard theory is actually true, RLHF-like processes are the only known generators of human-compatible values ever, and I’d update against the alignment theory / reasoning which called RLHF flawed. (Of course, there are reasons—like inductive biases—that RLHF-like processes could work in humans but not in AI, but any argument against RLHF would have to discriminate between the human/AI case in a way which accounts for those obstructions.)
On the object level:
If there’s some experience that deceives people to provide feedback signals that the behavior was prosocial, then it seems like the shard that leads to that experience will be reinforced.
Are you saying “The AI says something which makes us erroneously believe it saved a person’s life, and we reward it, and this can spawn a deception-shard”? If so—that’s not (necessarily) how credit assignment works. The AI’s credit assignment isn’t necessarily running along the lines of “people were deceived, so upweight computations which deceive people.”
Perhaps the AI thought the person would approve of that statement, and so it did it, and got rewarded, which reinforces the approval-seeking shard? (Which is bad news in a different way!)
Perhaps the AI was just exploring into a statement suggested by a self-supervised pretrained initialization, off of an already-learned general heuristic of “sometimes emit completions from the self-supervised pretrained world model.” Then the AI reinforces this heuristic (among other changes from the gradient).
the most logical reinforcement-based reason I can see why it doesn’t become a bigger problem is that people cannot reliably deceive each other.
I don’t know whether this is a problem at all, in general. I expect unaligned models to convergently deceive us. But that requires them to already be unaligned.
it’s just a huge gap from “Rewards have unpredictable effects on agent’s cognition, not necessarily to cause them to want reward” to “we have a way to use RL to interpret and implement human wishes.”
So, OP said
In general, we have no way to use RL to actually interpret and implement human wishes, rather than to optimize some concrete and easily-calculated reward signal.
I read into this a connotation of “In general, there isn’t a practically-findable way to use RL...”. I’m now leaning towards my original interpretation being wrong—that you meant something more like “we don’t know how to use RL to actually interpret and implement human wishes” (which I agree with).
I agree that this need not produce a policy that is trying to get reward, just one that in fact gets a lot of reward on distribution.
I think this tells us relatively little about the internal cognition, and so is a relatively non-actionable fact (which you probably agree with?). But I want to sort out my thoughts more, here, before laying down more of my intuitions on that.
Related clarifying question:
To illustrate how this can go wrong, imagine using RL to implement a decentralized autonomous organization (DAO) which maximizes its profit[...]
The shareholders of such a DAO may be able to capture the value it creates as long as they are able to retain effective control over its computing hardware / reward signal.
Do you remember why you thought it important that the shareholders retain control over the reward signal? I can think of several interpretations, but am curious what your take was.
similar to how if you read about enough social hacks you’ll probably be a bit scammy even tho you like people and don’t want to scam them
IDK if this is causally true or just evidentially true. I also further don’t know why it would be mechanistically relevant to the heuristic you posit.
Rather, I think that agents might end up with this heuristic at first, but over time it would get refined into “try new things which [among other criteria] aren’t obviously going to cause bad value drift away from current values.” One reason I expect the refinement in humans is that noticing your values drifted in a bad way is probably a negative reinforcement event, and so enough exploration-caused negative events might cause credit assignment to refine the heuristic into the shape I listed. This would convergently influence agents to not be reward-optimal, even on known-reachable-states. (I’m not super confident in this particular story porting over to AI, but think it’s a plausible outcome.)
If that’s kind of heuristic is a major underpinning of what we call “curiosity” in humans, then that would explain why I am, in general, not curious about exploring a life of crime, but am curious about math and art and other activities which won’t cause bad value drift away from my current values.
In other words, the fact that prosocial algorithms in brains recursively accommodate the fact that prosocial algorithms in brains are sometimes suboptimal (e.g., the capacity to think thoughts like “loving one’s neighbor is a double-edged sword”) is, I claim, a highly desirable property of prosocial brain algorithms. This ability to critically inspect one’s own values may well be the most important prosocial algorithm to pin down!
I think this is probably a convergent result of shard ecosystems doing reflection and planning on themselves, and less a set of human-specific algorithms (although maybe you didn’t mean to claim that). That is, I think that if you get the “base shards” ~right (i.e. they care about people in a variety of ways which I’m going to handwave for now, because I don’t know them more precisely yet), then the base shards will tend to end up doing value reflection. Another way of stating this position (originally argued to me by Quintin) is: “moral philosophy is weakly convergent in the type of its process (of reflecting on imperfections), but not in its reflective equilibria (i.e. the actual values which get settled on).”
Agree with the broader point of:
In essence, the ‘human moral reasoning ain’t all that’ critique ignores that human moral reasoning is itself responsible for generating this critique!
Also, see Quintin’s comment about how variance in human altruism is a good portent for alignment.
Being proven wrong is an awesome, exciting shortcut which lets me reach the truth even faster.