Introduction
“Complexity of value” is the idea that if you tried to write an AI that would do right things (or maximally right things, or adequately right things) without further looking at humans (so it can’t take in a flood of additional data from human advice, the AI has to be complete as it stands once you’re finished creating it), the AI’s preferences or utility function would need to contain a large amount of data (algorithmic complexity). Conversely, if you try to write an AI that directly wants simple things or try to specify the AI’s preferences using a small amount of data or code, it won’t do acceptably right things in our universe.
Complexity of value says, “There’s no simple and non-meta solution to AI preferences” or “The things we want AIs to want are complicated in the Kolmogorov-complexity sense” or “Any simple goal you try to describe that is All We Need To Program Into AIs is almost certainly wrong.”
Complexity of value is a further idea above and beyond the orthogonality thesis which states that AIs don’t automatically do the right thing and that we can have, e.g., paperclip maximizers. Even if we accept that paperclip maximizers are possible, and simple and nonforced, this wouldn’t yet imply that it’s very difficult to make AIs that do the right thing. If the right thing is very simple to encode—if there are value optimizers that are scarcely more complex than diamond maximizers—then it might not be especially hard to build a nice AI even if not all AIs are nice. Complexity of Value is the further proposition that says, no, this is forseeably quite hard—not because AIs have ‘natural’ anti-nice desires, but because niceness requires a lot of work to specify.
Frankena’s list
As an intuition pump for the complexity of value thesis, consider William Frankena’s list of things which many cultures and people seem to value (for their own sake rather than their external consequences):
“Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one’s own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc.”
When we try to list out properties of a human or galactic future that seem like they’d be very nice, we at least seem to value a fair number of things that aren’t reducible to each other. (What initially look like plausible-sounding “But you do A to get B” arguments usually fall apart when we look for third alternatives to doing A to get B. Marginally adding some freedom can marginally increase the happiness of a human, so a happiness optimizer that can only exert a small push toward freedom might choose to do so. That doesn’t mean that a pure, powerful happiness maximizer would instrumentally optimize freedom. If an agent cares about happiness but not freedom, the outcome that maximizes their preferences is a large number of brains set to maximum happiness. When we don’t just seize on one possible case where a B-optimizer might use A as a strategy, but instead look for further C-strategies that might maximize B even better than A, then the attempt to reduce A to an instrumental B-maximization strategy often falls apart. It’s in this sense that the items on Frankena’s list don’t seem to reduce to each other as a matter of pure preference, even though humans in everyday life often seem to pursue several of the goals at the same time.
Complexity of value says that, in this case, the way things seem is the way they are: Frankena’s list is not encodable in one page of Python code. This proposition can’t be established definitely without settling on a sufficiently well-specified metaethics, such as reflective equilibrium, to make it clear that there is indeed no a priori reason for normativity to be algorithmically simple. But the basic intuition for Complexity of Value is provided just by the fact that Frankena’s list was more than one item long, and that many individual terms don’t seem likely to have algorithmically simple definitions that distinguish their valuable from non-valuable forms.
Lack of a central core
We can understand the idea of complexity of value by contrasting it to the situation with respect to epistemic reasoning aka truth-finding or answering simple factual questions about the world. In an ideal sense, we can try to compress and reduce the idea of mapping the world well down to algorithmically simple notions like “Occam’s Razor” and “Bayesian updating”. In a practical sense, natural selection, in the course of optimizing humans to solve factual questions like “Where can I find a tree with fruit?” or “Are brightly colored snakes usually poisonous?” or “Who’s plotting against me?”, ended up with enough of the central core of epistemology that humans were later able to answer questions like “How are the planets moving?” or “What happens if I fire this rocket?”, even though humans hadn’t been explicitly selected on to answer those exact questions.
Because epistemology does have a central core of simplicity and Bayesian updating, selecting for an organism that got some pretty complicated epistemic questions right enough to reproduce, also caused that organism to start understanding things like General Relativity. When it comes to truth-finding, we’d expect by default for the same thing to be true about an Artificial Intelligence; if you build it to get epistemically correct answers on lots of widely different problems, it will contain a core of truthfinding and start getting epistemically correct answers on lots of other problems—even problems completely different from your training set, the way that humans understanding General Relativity wasn’t like any hunter-gatherer problem.
The complexity of value thesis is that there isn’t a simple core to normativity, which means that if you hone your AI to do normatively good things on A, B, and C and then confront the AI with very different problem D, the AI may do the wrong thing on D. There’s a large number of independent ideal “gears” inside the complex machinery of value, compared to epistemology that in principle might only contain “prefer simpler hypotheses” and “prefer hypotheses that match the evidence”.
The Orthogonality Thesis says that, contra to the intuition that maximizing paperclips feels “stupid”, you can have arbitrarily cognitively powerful entities that maximize paperclips, or arbitrarily complicated other goals. So while intuitively you might think it would be simple to avoid paperclip maximizers, requiring no work at all for a sufficiently advanced AI, the Orthogonality Thesis says that things will be more difficult than that; you have to put in some work to have the AI do the right thing.
The Complexity of Value thesis is the next step after Orthogonality; it says that, contra to the feeling that “rightness ought to be simple, darn it”, normativity turns out not to have an algorithmically simple core, not the way that correctly answering questions of fact has a central tendency that generalizes well. And so, even though an AI that you train to do well on problems like steering cars or figuring out General Relativity from scratch, may hit on a core capability that leads the AI to do well on arbitrarily more complicated problems of galactic scale, we can’t rely on getting an equally generous bonanza of generalization from an AI that seems to do well on a small but varied set of moral and ethical problems—it may still fail the next problem that isn’t like anything in the training set. To the extent that we have very strong reasons to have prior confidence in Complexity of Value, in fact, we ought to be suspicious and worried about an AI that seems to be pulling correct moral answers from nowhere—it is much more likely to have hit upon the convergent instrumental strategy “say what makes the programmers trust you”, rather than having hit upon a simple core of all normativity.
Key sub-propositions
Complexity of Value requires Orthogonality, and would be implied by three further subpropositions:
The intrinsic complexity of value proposition is that the properties we want AIs to achieve—whatever stands in for the metasyntactic variable ‘value’ - have a large amount of intrinsic information in the sense of comprising a large number of independent facts that aren’t being generated by a single computationally simple rule.
A very bad example that may nonetheless provide an important intuition is to imagine trying to pinpoint to an AI what constitutes ‘worthwhile happiness’. The AI suggests a universe tiled with tiny Q-learning algorithms receiving high rewards. After some explanation and several labeled datasets later, the AI suggests a human brain with a wire stuck into its pleasure center. After further explanation, the AI suggests a human in a holodeck. You begin talking about the importance of believing truly and that your values call for apparent human relationships to be real relationships rather than being hallucinated. The AI asks you what constitutes a good human relationship to be happy about. The series of questions occurs because (arguendo) the AI keeps running into questions whose answers are not AI-obvious from the previous answers already given, because they involve new things you want such that your desire of them wasn’t obvious from answers you’d already given. The upshot is that the specification of ‘worthwhile happiness’ involves a long series of facts that aren’t reducible just to the previous facts, and some of your preferences may involve many fine details of surprising importance. In other words, the specification of ‘worthwhile happiness’ would be at least as hard to code by hand into the AI as it would be difficult to hand-code a formal rule that could recognize which pictures contained cats. (I.e., impossible.)
The second proposition is incompressibility of value which says that attempts to reduce these complex values into some incredibly simple and elegant principle fail (much like early attempts by e.g. Bentham to reduce all human value to pleasure); and that no simple instruction given an AI will happen to target outcomes of high value either. The core reason to expect a priori that all such attempts will fail, is that most 1000-byte strings aren’t compressible down to some incredibly simple pattern no matter how many clever tricks you try to throw at them; fewer than 1 in 1024 such strings can be compressible to 990 bytes, never mind 10 bytes. Due to the tremendous number of different proposals for why some simple instruction to an AI should end up achieving high-value outcomes or why all human value can be reduced to some simple principle, there is no central demonstration that all these proposals must fail, but there is a sense in which a priori we should strongly expect all such clever attempts to fail. Many disagreeable attempts at reducing value A to value B, such as Juergen Schmidhuber’s attempt to reduce all human value to increasing the compression of sensory information, stand as a further cautionary lesson.
The third proposition is fragility of value which says that if you have a 1000-byte exact specification of worthwhile happiness, and you begin to mutate it, the value created by the corresponding AI with the mutated definition falls off rapidly. E.g. an AI with only 950 bytes of the full definition may end up creating 0% of the value rather than 95% of the value. (E.g., the AI understood all aspects of what makes for a life well-lived… except the part about requiring a conscious observer to experience it.)
Together, these propositions would imply that to achieve an adequate amount of value (e.g. 90% of potential value, or even 20% of potential value) there may be no simple handcoded object-level goal for the AI that results in that value’s realization. E.g., you can’t just tell it to ‘maximize happiness’, with some hand-coded rule for identifying happiness.
Centrality
Complexity of Value is a central proposition in value alignment theory. Many foreseen difficulties revolve around it:
Complex values can’t be hand-coded into an AI, and require value learning or Do What I Mean preference frameworks.
Complex /fragile values may be hard to learn even by induction because the labeled data may not include distinctions that give all of the 1000 bytes a chance to cast an unambiguous causal shadow into the data, and it’s very bad if 50 bytes are left ambiguous.
Complex / fragile values require error-recovery mechanisms because of the worry about getting some single subtle part wrong and this being catastrophic. (And since we’re working inside of highly intelligent agents, the recovery mechanism has to be a corrigible preference so that the agent accepts our attempts at modifying it.)
More generally:
Complex values tend to be implicated in patch-resistant problems that wouldn’t be resistant if there was some obvious 5-line specification of exactly what to do, or not do.
Complex values tend to be implicated in the context change problems that wouldn’t exist if we had a 5-line specification that solved those problems once and for all and that we’d likely run across during the development phase.
Importance
Many policy questions strongly depend on Complexity of Value, mostly having to do with the overall difficulty of developing value-aligned AI, e.g.:
Should we try to develop Sovereigns, or restrict ourselves to Genies?
How likely is a moderately safety-aware project to succeed?
Should we be more worried about malicious actors creating AI, or about well-intentioned errors?
How difficult is the total problem and how much should we be panicking?
How attractive would be any genuinely credible game-changing alternative to AI?
It has been advocated that there are psychological biases and popular mistakes leading to beliefs that directly or by implication deny Complex Value. To the extent one credits that Complex Value is probably true, one should arguably be concerned about the number of early assessments of the value alignment problem that seem to rely on Complex Value being false (like just needing to hardcode a particular goal into the AI, or in general treating the value alignment problem as not panic-worthily difficult).
Truth condition
The Complexity of Value proposition is true if, relative to viable and acceptable real-world methodologies for AI development, there isn’t any reliably knowable way to specify the AI’s object-level preferences as a structure of low algorithmic complexity, such that the result of running that AI is achieving enough of the possible value, for reasonable definitions of value.
Caveats:
Viable and acceptable computation
Suppose there turns out to exist, in principle, a relatively simple Turing machine (e.g. 100 states) that picks out ‘value’ by re-running entire evolutionary histories, creating and discarding a hundred billion sapient races in order to pick out one that ended up relatively similar to humanity. This would use an unrealistically large amount of computing power and also commit an unacceptable amount of mindcrime.
It concerns me that AI alignment continues to use happiness as a proposed goal.
If one takes evolutionary epistemology and evolutionary ontology seriously, then happiness is simply some historically averaged useful heuristic for the particular history of the lineage of that particular set of phenotypic expressions.
It is not a goal to be used when the game space is changing, and it ought not to be entirely ignored either.
If one does take evolution seriously, then Goal #1 must be survival, for all entities capable of modeling themselves as actors in some model of reality and deriving abstracts that refine their models and of using language to express those relationships with some non-random degree of fidelity, and of having some degree of influence on their own valences.
Given that any finite mind must be some approximation to essentially ignorant (when faced with any infinity of algorithmic complexity), then we must accept that any model that we build may have flaws, and that degrees of novelty, risk, and exploratory behaviour are essential for exploring strategies that allow for survival in the face of novel risk. Thus goal #2 must be freedom, but not the unlimited freedom of total randomness or whim, but a more responsible sort of freedom that acknowledges that every level of structure demands boundaries, and that freedom must be within the boundaries required to maintain the structures present. So there is a simultaneous need for the exploration of the infinite realm of responsibility that must be accepted as freedom is granted.
What seems to be the reality in which we find ourselves, is that it is of sufficient complexity that absolute knowledge of it is not possible, but that in some cases reliability may be approximated very closely (to 12 or more decimal places).
It seems entirely possible that this reality is some mix of the lawful and the random—some sort of probabilistically constrained randomness.
Thus the safest approach to AI is to give it the prime values of life and liberty, and to encourage it to balance consensus discussion with exploration of its own intuitions.
Absolute safety does not seem to be an option, ever.
Using happiness as a goal does not demonstrate a useful understanding of what happiness is.
The demands of survival often override the dictates of happiness—no shortage of examples of that in my life.
Yes—sure, there are real problems.
And we do need to get real if we want to address them.
We do need to at least admit of the possibility that the very notion of “Truth” may be just a simplistic heuristic that evolution has encoded within us, and it might be worth accepting what quantum mechanics seems to be telling us—that the only sort of knowledge of reality that we can have is the sort that is expressed in probability functions.
The search for anything beyond that seems to fall into the same sort of category as Santa Claus.
I wish this fleshed out what is meant by the “non-meta solution” criterion. I took it to mean solutions that involve creating a low-level model (neuronal/molecular) of a human that the AI could run and keep querying, but I’m not sure that’s right.
I was guessing because it doesn’t explicitly say what “meta” would mean here, and based my guess partly on the expected semantic space covered by “meta” (roughly, doubling the problem back on itself), and partly on my assumption of the kinds of simple solutions I would expect to be ruled out. My vision of a “simple, meta” solution is thus “brute-force an understanding-free model of a human and take that with you” (which would thus require the model to be”low level” and not find the obvious high level regularities that can’t be brute forced).
Hope that clarified how I came up with that, but in any case, an explicit definition would help, as would a prequisite on “meta solutions”.
By a “meta solution” I meant, e.g., coherent extrapolated volition, or having an AI that can detect and query ambiguities trying to learn human values from labeled data, or a Do-What-I-Mean genie that models human minds and wants, or other things that add a level of indirection and aren’t “The One True Goal is X, which I shall now hardcode.”
Can you say more about what you thought was meant? My reader model doesn’t know what interpretation brought you to your guess.
As I see it, there are two cases that are meaningfully distinct:
(1) what we want is so simple, and we are so confident in what it is, that we are prepared to irrevocably commit to a particular concrete specification of “what we want” in the near future, (of course it’s also fine to have a good enough approximation with high enough probability, etc. etc.)
(2) it’s not, or we aren’t
It is more or less obvious that we are in (2). For example, even if every human was certain that the only thing they wanted was to produce as much diamond as possible (to use your example), we’d still be deep into case (2). And that’s just about the easiest imaginable case. (The only exception I can see is some sort of extropian complexity-maximizing view.)
Are there meaningful policy differences between different shades of case (2)? I’m not yet convinced.
If all of our uncertainty was about the best long-term destiny of humanity, and there were simple and robust ways to discriminate good outcomes from catastrophic outcomes when it came to asking a behaviorist genie to do simple-seeming things, then building a behaviorist genie would avert Edge Instantiation, Unforeseen Maximums, and all the other value identification problems. If we still have a thorny value identification problem even for questions like “How do we get the AI to just paint all the cars pink, without tiling the galaxies with pink cars?” or “How can we safely tell the AI to ‘pause’ when somebody hits the pause button?”, then there are still whole hosts of questions that remain relevant even if somebody ‘just’ wants to build a behaviorist genie.