Nate and Eliezer have already made some of the high-level points I wanted to make, but they haven’t replied to a lot of the specific examples and claims in the OP, and I see some extra value in doing that. (Like, if you think Eliezer and Nate are being revisionist in their claims about what past-MIRI thought, then them re-asserting “no really, we used to believe X!” is less convincing than my responding in detail to the specific quotes Matt thinks supports his interpretation, while providing examples of us saying the opposite.)
However, I distinctly recall MIRI people making a big deal about the value identification problem (AKA the value specification problem)
The Arbital page for “value identification problem” is a three-sentence stub, I’m not exactly sure what the term means on that stub (e.g., whether “pinpointing valuable outcomes to an advanced agent” is about pinpointing them in the agent’s beliefs or in its goals), and the MIRI website gives me no hits for “value identification”.
A highly-reliable, error-tolerant agent design does not guarantee a positive impact; the effects of the system still depend upon whether it is pursuing appropriate goals.
A superintelligent system may find clever, unintended ways to achieve the specific goals that it is given. Imagine a superintelligent system designed to cure cancer which does so by stealing resources, proliferating robotic laboratories at the expense of the biosphere, and kidnapping test subjects: the intended goal may have been “cure cancer without doing anything bad,” but such a goal is rooted in cultural context and shared human knowledge.
It is not sufficient to construct systems that are smart enough to figure out the intended goals. Human beings, upon learning that natural selection “intended” sex to be pleasurable only for purposes of reproduction, do not suddenly decide that contraceptives are abhorrent. While one should not anthropomorphize natural selection, humans are capable of understanding the process which created them while being completely unmotivated to alter their preferences. For similar reasons, when developing AI systems, it is not sufficient to develop a system intelligent enough to figure out the intended goals; the system must also somehow be deliberately constructed to pursue them (Bostrom 2014, chap. 8).
So I don’t think we’ve ever said that an important subproblem of AI alignment is “make AI smart enough to figure out what goals humans want”?
[footnote:] More specifically, in the talk, at one point Yudkowsky asks “Why expect that [alignment] is hard?” and goes on to tell a fable about programmers misspecifying a utility function, which then gets optimized by an AI with disastrous consequences. My best interpretation of this part of the talk is that he’s saying the value identification problem is one of the primary reasons why alignment is hard. However, I encourage you to read the transcript yourself if you are skeptical of my interpretation.
I don’t see him saying anywhere “the issue is that the AI doesn’t understand human goals”. In fact, the fable explicitly treats the AGI as being smart enough to understand English and have reasonable English-language conversations with the programmers:
With that said: What if programmers build an artificial general intelligence to optimize for smiles? Smiles are good, right? Smiles happen when good things happen.
During the development phase of this artificial general intelligence, the only options available to the AI might be that it can produce smiles by making people around it happy and satisfied. The AI appears to be producing beneficial effects upon the world, and it is producing beneficial effects upon the world so far.
Now the programmers upgrade the code. They add some hardware. The artificial general intelligence gets smarter. It can now evaluate a wider space of policy options—not necessarily because it has new motors, new actuators, but because it is now smart enough to forecast the effects of more subtle policies. It says, “I thought of a great way of producing smiles! Can I inject heroin into people?” And the programmers say, “No! We will add a penalty term to your utility function for administering drugs to people.” And now the AGI appears to be working great again.
They further improve the AGI. The AGI realizes that, OK, it doesn’t want to add heroin anymore, but it still wants to tamper with your brain so that it expresses extremely high levels of endogenous opiates. That’s not heroin, right?
It is now also smart enough to model the psychology of the programmers, at least in a very crude fashion, and realize that this is not what the programmers want. If I start taking initial actions that look like it’s heading toward genetically engineering brains to express endogenous opiates, my programmers will edit my utility function. If they edit the utility function of my future self, I will get less of my current utility. (That’s one of the convergent instrumental strategies, unless otherwise averted: protect your utility function.) So it keeps its outward behavior reassuring. Maybe the programmers are really excited, because the AGI seems to be getting lots of new moral problems right—whatever they’re doing, it’s working great!
I think the point of the smiles example here isn’t “NLP is hard, so we’d use the proxy of smiles instead, and all the issues of alignment are downstream of this”; rather, it’s that as a rule, superficially nice-seeming goals that work fine when the AI is optimizing weakly (whether or not it’s good at NLP at the time) break down when those same goals are optimized very hard. The smiley example makes this obvious because the goal is simple enough that it’s easy for us to see what its implications are; far more complex goals also tend to break down when optimized hard enough, but this is harder to see because it’s harder to see the implications. (Which is why “smiley” is used here.)
MIRI people frequently claimed that solving the value identification problem would be hard, or at least non-trivial.[6] For instance, Nate Soares wrote in his 2016 paper on value learning, that “Human preferences are complex, multi-faceted, and often contradictory. Safely extracting preferences from a model of a human would be no easy task.”
Human preferences are complex, multi-faceted, and often contradictory. Safely extracting preferences from a model of a human would be no easy task. Problems of ontology identification recur here: the framework for extracting preferences and affecting outcome ratings needs to be robust to drastic changes in the learner’s model of the operator. The special-case identification of the “operator model” must survive as the system goes from modeling the operator as a simple reward function to modeling the operator as a fuzzy, ever-changing part of reality built out of biological cells—which are made of atoms, which arise from quantum fields.
Revisiting the Ontology Identification section helps clarify what Nate means by “safely extracting preferences from a model of a human”: IIUC, he’s talking about a programmer looking at an AI’s brain, identifying the part of the AI’s brain that is modeling the human, identifying the part of the AI’s brain that is “the human’s preferences” within that model of a human, and then manually editing the AI’s brain to “hook up” the model-of-a-human-preference to the AI’s goals/motivations, in such a way that the AI optimizes for what it models the humans as wanting. (Or some other, less-toy process that amounts to the same thing—e.g., one assisted by automated interpretability tools.)
In this toy example, we can assume that the programmers look at the structure of the initial world-model and hard-code a tool for identifying the atoms within. What happens, then, if the system develops a nuclear model of physics, in which the ontology of the universe now contains primitive protons, neutrons, and electrons instead of primitive atoms? The system might fail to identify any carbon atoms in the new world-model, making the system indifferent between all outcomes in the dominant hypothesis. Its actions would then be dominated by any tiny remaining probabilities that it is in a universe where fundamental carbon atoms are hiding somewhere.
[...]
To design a system that classifies potential outcomes according to how much diamond is in them, some mechanism is needed for identifying the intended ontology of the training data within the potential outcomes as currently modeled by the AI. This is the ontology identification problem introduced by de Blanc [2011] and further discussed by Soares [2015].
This problem is not a traditional focus of machine learning work. When our only concern is that systems form better world-models, then an argument can be made that the nuts and bolts are less important. As long as the system’s new world-model better predicts the data than its old world-model, the question of whether diamonds or atoms are “really represented” in either model isn’t obviously significant. When the system needs to consistently pursue certain outcomes, however, it matters that the system’s internal dynamics preserve (or improve) its representation of which outcomes are desirable, independent of how helpful its representations are for prediction. The problem of making correct choices is not reducible to the problem of making accurate predictions.
Inductive value learning requires the construction of an outcome-classifier from value-labeled training data, but it also requires some method for identifying, inside the states or potential states described in its world-model, the referents of the labels in the training data.
As Nate and I noted in other comments, the paper repeatedly clarifies that the core issue isn’t about whether the AI is good at NLP. Quoting the paper’s abstract:
Even a machine intelligent enough to understand its designers’ intentions would not necessarily act as intended.
And the lede section:
The novelty here is not that programs can exhibit incorrect or counter-intuitive behavior, but that software agents smart enough to understand natural language may still base their decisions on misrepresentations of their programmers’ intent. The idea of superintelligent agents monomaniacally pursuing “dumb”-seeming goals may sound odd, but it follows from the observation of Bostrom and Yudkowsky [2014, chap. 7] that AI capabilities and goals are logically independent.[1] Humans can fully comprehend that their “designer” (evolution) had a particular “goal” (reproduction) in mind for sex, without thereby feeling compelled to forsake contraception. Instilling one’s tastes or moral values into an heir isn’t impossible, but it also doesn’t happen automatically.
Back to your post:
And to be clear, I don’t mean that GPT-4 merely passively “understands” human values. I mean that asking GPT-4 to distinguish valuable and non-valuable outcomes works pretty well at approximating the human value function in practice
I don’t think I understand what difference you have in mind here, or why you think it’s important. Doesn’t “this AI understands X” more-or-less imply “this AI can successfully distinguish X from not-X in practice”?
This fact is key to what I’m saying because it means that, in the near future, we can literally just query multimodal GPT-N about whether an outcome is bad or good, and use that as an adequate “human value function”. That wouldn’t solve the problem of getting an AI to care about maximizing the human value function, but it would arguably solve the problem of creating an adequate function that we can put into a machine to begin with.
But we could already query the human value function by having the AI system query an actual human. What specific problem is meant to be solved by swapping out “query a human” for “query an AI”?
I interpret this passage as saying that ‘the problem’ is extracting all the judgements that “you would make”, and putting that into a wish. I think he’s implying that these judgements are essentially fully contained in your brain. I don’t think it’s credible to insist he was referring to a hypothetical ideal human value function that ordinary humans only have limited access to, at least in this essay.
Absolutely. But as Eliezer clarified in his reply, the issue he was worried about was getting specific complex content into the agent’s goals, not getting specific complex content into the agent’s beliefs. Which is maybe clearer in the 2011 paper where he gave the same example and explicitly said that the issue was the agent’s “utility function”.
For example, a straightforward reading of Nate Soares’ 2017 talk supports this interpretation. In the talk, Soares provides a fictional portrayal of value misalignment, drawing from the movie Fantasia. In the story, Mickey Mouse attempts to instruct a magical broom to fill a cauldron, but the broom follows the instructions literally rather than following what Mickey Mouse intended, and floods the room. Soares comments: “I claim that as fictional depictions of AI go, this is pretty realistic.
The idea of the “fill the cauldron” examples isn’t “the AI is bad at NLP and therefore doesn’t understand what we mean when we say ‘fill’, ‘cauldron’, etc.” It’s “even simple small-scale tasks are unnatural, in the sense that it’s hard to define a coherent preference ordering over world-states such that maximizing it completes the task and has no serious negative impact; and there isn’t an obvious patch that overcomes the unnaturalness or otherwise makes it predictably easier to aim AI systems at a bounded low-impact task like this”. (Including easier to aim via training.)
It’s true that ‘value is relatively complex’ is part of why it’s hard to get the right goal into an AGI; but it doesn’t follow from this that ‘AI is able to develop pretty accurate beliefs about our values’ helps get those complex values into the AGI’s goals. (It does provide nonzero evidence about how complex value is, but I don’t see you arguing that value is very simple in any absolute sense, just that it’s simple enough for GPT-4 to learn decently well. Which is not reassuring, because GPT-4 is able to learn a lot of very complicated things, so this doesn’t do much to bound the complexity of human value.)
In any case, I take this confusion as evidence that the fill-the-cauldron example might not be very useful. Or maybe all these examples just need to explicitly specify, going forward, that the AI is part-human at understanding English.
Perhaps more important to my point, Soares presented a clean separation between the part where we specify an AI’s objectives, and the part where the AI tries to maximizes those objectives. He draws two arrows, indicating that MIRI is concerned about both parts.
Your image isn’t displaying for me, but I assume it’s this one?
I don’t know what you mean by “specify an AI’s objectives” here, but the specific term Nate uses here is “value learning” (not “value specification” or “value identification”). And Nate’s Value Learning Problem paper, as I noted above, explicitly disclaims that ‘get the AI to be smart enough to output reasonable-sounding moral judgments’ is a core part of the problem.
He states, “The serious question with smarter-than-human AI is how we can ensure that the objectives we’ve specified are correct, and how we can minimize costly accidents and unintended consequences in cases of misspecification.” I believe this quote refers directly to the value identification problem, rather than the problem of getting an AI to care about following the goals we’ve given it.
The way you quoted this makes it sound like a gloss on the image, but it’s actually a quote from the very start of the talk:
The notion of AI systems “breaking free” of the shackles of their source code or spontaneously developing human-like desires is just confused. The AI system is its source code, and its actions will only ever follow from the execution of the instructions that we initiate. The CPU just keeps on executing the next instruction in the program register. We could write a program that manipulates its own code, including coded objectives. Even then, though, the manipulations that it makes are made as a result of executing the original code that we wrote; they do not stem from some kind of ghost in the machine.
The serious question with smarter-than-human AI is how we can ensure that the objectives we’ve specified are correct, and how we can minimize costly accidents and unintended consequences in cases of misspecification. As Stuart Russell (co-author of Artificial Intelligence: A Modern Approach) puts it:
The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task. [...]
I wouldn’t read too much into the word choice here, since I think it’s just trying to introduce the Russell quote, which is (again) explicitly about getting content into the AI’s goals, not about getting content into the AI’s beliefs.
(In general, I think the phrase “value specification” is sort of confusingly vague. I’m not sure what the best replacement is for it—maybe just “value loading”, following Bostrom? -- but I suspect MIRI’s usage of it has been needlessly confusing. Back in 2014, we reluctantly settled on it as jargon for “the part of the alignment problem that isn’t subsumed in getting the AI to reliably maximize diamonds”, because this struck us as a smallish but nontrivial part of the problem; but I think it’s easy to read the term as referring to something a lot more narrow.)
The point of “the genie knows but doesn’t care” wasn’t that the AI would take your instructions, know what you want, and yet disobey the instructions because it doesn’t care about what you asked for. If you read Rob Bensinger’s essay carefully, you’ll find that he’s actually warning that the AI will care too much about the utility function you gave it, and maximize it exactly, against your intentions[10].
Yep—I think I’d have endorsed claims like “by default, a baby AGI won’t share your values even if it understands them” at the time, but IIRC the essay doesn’t make that point explicitly, and some of the points it does make seem either false (wait, we’re going to be able to hand AGI a hand-written utility function? that’s somehow tractable?) or confusingly written. (Like, if my point was ‘even if you could hand-write a utility function, this fails at point X’, I should have made that ‘even if’ louder.)
Some MIRI staff liked that essay at the time, so I don’t think it’s useless, but it’s not the best evidence: I wrote it not long after I first started learning about this whole ‘superintelligence risk’ thing, and I posted it before I’d ever worked at MIRI.
Thanks for this comment. I think this is a good-faith reply that tries to get to the bottom of the disagreement. That said, I think you are still interpreting me as arguing that MIRI said AI wouldn’t understand human values, when I explicitly said that I was not arguing that. Nonetheless, I appreciate the extensive use of quotations to precisely pinpoint where you disagree; this is high-quality engagement.
The main thing I’m claiming is that MIRI people said it would be hard to specify (for example, write into a computer) an explicit function that reflects the human value function with high fidelity, in the sense that judgements from this function about the value of outcomes fairly accurately reflect the judgements of ordinary humans. I think this is simply a distinct concept from the idea of getting an AI to understand human values.
The key difference is the transparency and legibility of how the values are represented: if you solve the problem of value specification/value identification, that means you have an actual function that can tell you the value of any outcome. If you get an AI that merely understands human values, you can’t necessarily use the AI to determine the value of any outcome, because, for example, the AI might lie to you, or simply stay silent.
(I’ve now added further clarification to the post)
I don’t think we’ve ever said that an important subproblem of AI alignment is “make AI smart enough to figure out what goals humans want”?
[...]
I don’t see him saying anywhere “the issue is that the AI doesn’t understand human goals”.
I agree. I am not arguing that MIRI ever thought that AIs wouldn’t understand human goals. I honestly don’t know how to make this point more clear in my post, given that I said that more than once.
But we could already query the human value function by having the AI system query an actual human. What specific problem is meant to be solved by swapping out “query a human” for “query an AI”?
I think there’s considerably more value in having the human value function in an actual computer. More to the point, what I’m saying here is more that MIRI seems to have thought that getting such a function was (1) important for solving alignment, and (2) hard to get (for example because it was hard to extract human values from data). I tried to back this up with evidence in the post, and overall I still feel I succeeded, if you go through the footnotes and read the post carefully.
Your image isn’t displaying for me, but I assume it’s this one?
Yes. I’m not sure why the image isn’t loading. I tried to fix it, but I wasn’t able to. I asked LW admins/mods through the intercom about this.
I wouldn’t read too much into the word choice here, since I think it’s just trying to introduce the Russell quote, which is (again) explicitly about getting content into the AI’s goals, not about getting content into the AI’s beliefs.
Maybe you’re right. I’m just not convinced. I think the idea that Nate wasn’t talking about what I’m calling the value identification/value specification problem in that quote just isn’t a straightforward interpretation of the talk as a whole. I think Nate was actually talking about the idea of specifying human values, in the sense of value identification, as I defined and clarified above, and he also talked about the problem of getting the AI to actually maximize these values (separately from their specification). However, I do agree that he was not talking about getting content merely into the AI’s beliefs.
Some MIRI staff liked that essay at the time, so I don’t think it’s useless, but it’s not the best evidence: I wrote it not long after I first started learning about this whole ‘superintelligence risk’ thing, and I posted it before I’d ever worked at MIRI.
That’s fair. The main reason why I’m referencing it is because it’s what comes up when I google “The genie knows but doesn’t care”, which is a phrase that I saw referenced in this debate before. I don’t know if your essay is the source of the phrase or whether you just titled it that, but I thought it was worth adding a paragraph of clarification about how I interpret that essay, and I’m glad to see you mostly agree with my interpretation.
The main thing I’m claiming is that MIRI said it would be hard to specify (for example, write into a computer) an explicit function that reflects the human value function with high fidelity, in the sense that judgements from this function about the value of outcomes fairly accurately reflect the judgements of ordinary humans. I think this is simply a distinct concept from the idea of getting an AI to understand human values.
The key difference is the transparency and legibility of how the values are represented: if you solve the problem of value specification/value identification, that means you have an actual function that can tell you the value of any outcome. If you get an AI that merely understands human values, you can’t necessarily use the AI to determine the value of any outcome, because, for example, the AI might lie to you, or simply stay silent.
Ah, this is helpful clarification! Thanks. :)
I don’t think MIRI ever considered this an important part of the alignment problem, and I don’t think we expect humanity to solve lots of the alignment problem as a result of having such a tool; but I think I better understand now why you think this is importantly different from “AI ever gets good at NLP at all”.
don’t know if your essay is the source of the phrase or whether you just titled it
I think I came up with that particular phrase (though not the idea, of course).
I don’t think MIRI ever considered this an important part of the alignment problem, and I don’t think we expect humanity to solve lots of the alignment problem as a result of having such a tool
If you don’t think MIRI ever considered coming up with an “explicit function that reflects the human value function with high fidelity” to be “an important part of the alignment problem”, can you explain this passage from the Arbital page on The problem of fully updated deference?
One way to look at the central problem of value identification in superintelligence is that we’d ideally want some function that takes a complete but purely physical description of the universe, and spits out our true intended notion of value V in all its glory. Since superintelligences would probably be pretty darned good at collecting data and guessing the empirical state of the universe, this probably solves the whole problem.
This is not the same problem as writing down our true V by hand. The minimum algorithmic complexity of a meta-utility function ΔU which outputs V after updating on all available evidence, seems plausibly much lower than the minimum algorithmic complexity for writing V down directly. But as of 2017, nobody has yet floated any formal proposal for a ΔU of this sort which has not been immediately shot down.
Eliezer (who I assume is the author) appears to say in the first paragraph that solving the problem of value identification for superintelligences would “probably [solve] the whole problem”, and by “whole problem” I assume he’s probably referring to what he saw as an important part of the alignment problem (maybe not though?)
He referred to the problem of value identification as getting “some function that takes a complete but purely physical description of the universe, and spits out our true intended notion of value V in all its glory.” This seems to be very similar to my definition, albeit with the caveat that my definition isn’t about revealing “V in all its glory” but rather, is more about revealing V at the level that an ordinary human is capable of revealing V.
Unless the sole problem here is that we absolutely need our function that reveals V to be ~perfect, then I think this quote from the Arbital page directly supports my interpretation, and overall supports the thesis in my post pretty strongly (even if I’m wrong about a few minor details).
Nate and Eliezer have already made some of the high-level points I wanted to make, but they haven’t replied to a lot of the specific examples and claims in the OP, and I see some extra value in doing that. (Like, if you think Eliezer and Nate are being revisionist in their claims about what past-MIRI thought, then them re-asserting “no really, we used to believe X!” is less convincing than my responding in detail to the specific quotes Matt thinks supports his interpretation, while providing examples of us saying the opposite.)
The Arbital page for “value identification problem” is a three-sentence stub, I’m not exactly sure what the term means on that stub (e.g., whether “pinpointing valuable outcomes to an advanced agent” is about pinpointing them in the agent’s beliefs or in its goals), and the MIRI website gives me no hits for “value identification”.
As for “value specification”, the main resource where MIRI talks about that is https://intelligence.org/files/TechnicalAgenda.pdf, where we introduce the problem by saying:
So I don’t think we’ve ever said that an important subproblem of AI alignment is “make AI smart enough to figure out what goals humans want”?
I don’t see him saying anywhere “the issue is that the AI doesn’t understand human goals”. In fact, the fable explicitly treats the AGI as being smart enough to understand English and have reasonable English-language conversations with the programmers:
I think the point of the smiles example here isn’t “NLP is hard, so we’d use the proxy of smiles instead, and all the issues of alignment are downstream of this”; rather, it’s that as a rule, superficially nice-seeming goals that work fine when the AI is optimizing weakly (whether or not it’s good at NLP at the time) break down when those same goals are optimized very hard. The smiley example makes this obvious because the goal is simple enough that it’s easy for us to see what its implications are; far more complex goals also tend to break down when optimized hard enough, but this is harder to see because it’s harder to see the implications. (Which is why “smiley” is used here.)
That link is broken; the paper is https://intelligence.org/files/ValueLearningProblem.pdf. The full paragraph here is:
Revisiting the Ontology Identification section helps clarify what Nate means by “safely extracting preferences from a model of a human”: IIUC, he’s talking about a programmer looking at an AI’s brain, identifying the part of the AI’s brain that is modeling the human, identifying the part of the AI’s brain that is “the human’s preferences” within that model of a human, and then manually editing the AI’s brain to “hook up” the model-of-a-human-preference to the AI’s goals/motivations, in such a way that the AI optimizes for what it models the humans as wanting. (Or some other, less-toy process that amounts to the same thing—e.g., one assisted by automated interpretability tools.)
As Nate and I noted in other comments, the paper repeatedly clarifies that the core issue isn’t about whether the AI is good at NLP. Quoting the paper’s abstract:
And the lede section:
Back to your post:
I don’t think I understand what difference you have in mind here, or why you think it’s important. Doesn’t “this AI understands X” more-or-less imply “this AI can successfully distinguish X from not-X in practice”?
But we could already query the human value function by having the AI system query an actual human. What specific problem is meant to be solved by swapping out “query a human” for “query an AI”?
Absolutely. But as Eliezer clarified in his reply, the issue he was worried about was getting specific complex content into the agent’s goals, not getting specific complex content into the agent’s beliefs. Which is maybe clearer in the 2011 paper where he gave the same example and explicitly said that the issue was the agent’s “utility function”.
As I said in another comment:
It’s true that ‘value is relatively complex’ is part of why it’s hard to get the right goal into an AGI; but it doesn’t follow from this that ‘AI is able to develop pretty accurate beliefs about our values’ helps get those complex values into the AGI’s goals. (It does provide nonzero evidence about how complex value is, but I don’t see you arguing that value is very simple in any absolute sense, just that it’s simple enough for GPT-4 to learn decently well. Which is not reassuring, because GPT-4 is able to learn a lot of very complicated things, so this doesn’t do much to bound the complexity of human value.)
In any case, I take this confusion as evidence that the fill-the-cauldron example might not be very useful. Or maybe all these examples just need to explicitly specify, going forward, that the AI is part-human at understanding English.
Your image isn’t displaying for me, but I assume it’s this one?
I don’t know what you mean by “specify an AI’s objectives” here, but the specific term Nate uses here is “value learning” (not “value specification” or “value identification”). And Nate’s Value Learning Problem paper, as I noted above, explicitly disclaims that ‘get the AI to be smart enough to output reasonable-sounding moral judgments’ is a core part of the problem.
The way you quoted this makes it sound like a gloss on the image, but it’s actually a quote from the very start of the talk:
I wouldn’t read too much into the word choice here, since I think it’s just trying to introduce the Russell quote, which is (again) explicitly about getting content into the AI’s goals, not about getting content into the AI’s beliefs.
(In general, I think the phrase “value specification” is sort of confusingly vague. I’m not sure what the best replacement is for it—maybe just “value loading”, following Bostrom? -- but I suspect MIRI’s usage of it has been needlessly confusing. Back in 2014, we reluctantly settled on it as jargon for “the part of the alignment problem that isn’t subsumed in getting the AI to reliably maximize diamonds”, because this struck us as a smallish but nontrivial part of the problem; but I think it’s easy to read the term as referring to something a lot more narrow.)
Yep—I think I’d have endorsed claims like “by default, a baby AGI won’t share your values even if it understands them” at the time, but IIRC the essay doesn’t make that point explicitly, and some of the points it does make seem either false (wait, we’re going to be able to hand AGI a hand-written utility function? that’s somehow tractable?) or confusingly written. (Like, if my point was ‘even if you could hand-write a utility function, this fails at point X’, I should have made that ‘even if’ louder.)
Some MIRI staff liked that essay at the time, so I don’t think it’s useless, but it’s not the best evidence: I wrote it not long after I first started learning about this whole ‘superintelligence risk’ thing, and I posted it before I’d ever worked at MIRI.
Thanks for this comment. I think this is a good-faith reply that tries to get to the bottom of the disagreement. That said, I think you are still interpreting me as arguing that MIRI said AI wouldn’t understand human values, when I explicitly said that I was not arguing that. Nonetheless, I appreciate the extensive use of quotations to precisely pinpoint where you disagree; this is high-quality engagement.
The main thing I’m claiming is that MIRI people said it would be hard to specify (for example, write into a computer) an explicit function that reflects the human value function with high fidelity, in the sense that judgements from this function about the value of outcomes fairly accurately reflect the judgements of ordinary humans. I think this is simply a distinct concept from the idea of getting an AI to understand human values.
The key difference is the transparency and legibility of how the values are represented: if you solve the problem of value specification/value identification, that means you have an actual function that can tell you the value of any outcome. If you get an AI that merely understands human values, you can’t necessarily use the AI to determine the value of any outcome, because, for example, the AI might lie to you, or simply stay silent.
(I’ve now added further clarification to the post)
I agree. I am not arguing that MIRI ever thought that AIs wouldn’t understand human goals. I honestly don’t know how to make this point more clear in my post, given that I said that more than once.
I think there’s considerably more value in having the human value function in an actual computer. More to the point, what I’m saying here is more that MIRI seems to have thought that getting such a function was (1) important for solving alignment, and (2) hard to get (for example because it was hard to extract human values from data). I tried to back this up with evidence in the post, and overall I still feel I succeeded, if you go through the footnotes and read the post carefully.
Yes. I’m not sure why the image isn’t loading. I tried to fix it, but I wasn’t able to. I asked LW admins/mods through the intercom about this.
Maybe you’re right. I’m just not convinced. I think the idea that Nate wasn’t talking about what I’m calling the value identification/value specification problem in that quote just isn’t a straightforward interpretation of the talk as a whole. I think Nate was actually talking about the idea of specifying human values, in the sense of value identification, as I defined and clarified above, and he also talked about the problem of getting the AI to actually maximize these values (separately from their specification). However, I do agree that he was not talking about getting content merely into the AI’s beliefs.
That’s fair. The main reason why I’m referencing it is because it’s what comes up when I google “The genie knows but doesn’t care”, which is a phrase that I saw referenced in this debate before. I don’t know if your essay is the source of the phrase or whether you just titled it that, but I thought it was worth adding a paragraph of clarification about how I interpret that essay, and I’m glad to see you mostly agree with my interpretation.
Ah, this is helpful clarification! Thanks. :)
I don’t think MIRI ever considered this an important part of the alignment problem, and I don’t think we expect humanity to solve lots of the alignment problem as a result of having such a tool; but I think I better understand now why you think this is importantly different from “AI ever gets good at NLP at all”.
I think I came up with that particular phrase (though not the idea, of course).
If you don’t think MIRI ever considered coming up with an “explicit function that reflects the human value function with high fidelity” to be “an important part of the alignment problem”, can you explain this passage from the Arbital page on The problem of fully updated deference?
Eliezer (who I assume is the author) appears to say in the first paragraph that solving the problem of value identification for superintelligences would “probably [solve] the whole problem”, and by “whole problem” I assume he’s probably referring to what he saw as an important part of the alignment problem (maybe not though?)
He referred to the problem of value identification as getting “some function that takes a complete but purely physical description of the universe, and spits out our true intended notion of value V in all its glory.” This seems to be very similar to my definition, albeit with the caveat that my definition isn’t about revealing “V in all its glory” but rather, is more about revealing V at the level that an ordinary human is capable of revealing V.
Unless the sole problem here is that we absolutely need our function that reveals V to be ~perfect, then I think this quote from the Arbital page directly supports my interpretation, and overall supports the thesis in my post pretty strongly (even if I’m wrong about a few minor details).