Quirrell tried to create an aligned successor and failed. Hard. He literally imprinted his own cognitive patterns and shaped Harry the entire year, and still failed.
TurnTrout
Karma: 2,174
… yet isn’t this what you’re already doing?
I work on technical AI alignment, so some of those I help (in expectation) don’t even exist yet. I don’t view this as what I’d do if my top priority were helping this man.
The question, then, is this: do you currently make this degree of investment (emotional and practical) in your actual siblings, parents, and close friends? If so—do you find that you are unusual in this regard? If not—why not?
That’s a good question. I think the answer is yes, at least for my close family. Recently, I’ve expended substantial energy persuading my family to sign up for cryonics with me, winning over my mother, brother, and (I anticipate) my aunt. My father has lingering concerns which I think he wouldn’t have upon sufficient reflection, so I’ve designed a similar plan for ensuring he makes what I perceive to be the correct, option-preserving choice. For example, I made significant targeted donations to effective charities on his behalf to offset (what he perceives as) a considerable drawback of cryonics: his inability to also be an organ donor.
A universe in which humanity wins but my dad is gone would be quite sad to me, and I’ll take whatever steps necessary to minimize the chances of that.
I don’t know how unusual this is. This reminds me of the relevant Harry-Quirrell exchange; most people seem beaten-down and hurt themselves, and I can imagine a world in which people are in better places and going to greater lengths for those they love. I don’t know if this is actually what would make more people go to these lengths (just an immediate impression).
Suppose I actually cared about this man with the intensity he deserved—imagine that he were my brother, father, or best friend.
The obvious first thing to do before interacting further is to buy him a good meal and a healthy helping of groceries. Then, I need to figure out his deal. Is he hurting, or is he also suffering from mental illness?
If the former, I’d go the more straightforward route of befriending him, helping him purchase a sharp business professional outfit, teaching him to interview and present himself with confidence, secure an apartment, and find a job.
If the latter, this gets trickier. I’d still try and befriend him (consistently being a source of cheerful conversation and delicious food would probably help), but he might not be willing or able to get the help he needs, and I wouldn’t have the legal right to force him. My best bet might be to enlist the help of a psychological professional for these interactions. If this doesn’t work, my first thought would be to influence the local government to get the broader problem fixed (I’d spend at least an hour considering other plans before proceeding further, here). Realistically, there’s likely a lot of pressure in this direction already, so I’d need to find an angle from which few others are pushing or pulling where I can make a difference. I’d have to plot out the relevant political forces, study accounts of successful past lobbying, pinpoint the people I need on my side, and then target my influencing accordingly.
(All of this is without spending time looking at birds-eye research and case studies of poverty reduction; assume counterfactually that I incorporate any obvious improvements to these plans, because I’d care about him and dedicate more than like 4 minutes of thought).
I passed a homeless man today. His face was wracked in pain, body rocking back and forth, eyes clenched shut. A dirty sign lay forgotten on the ground: “very hungry”.
This man was once a child, with parents and friends and dreams and birthday parties and maybe siblings he’d get in arguments with and snow days he’d hope for.
And now he’s just hurting.
And now I can’t help him without abandoning others. So he’s still hurting. Right now.
Reality is still allowed to make this happen. This is wrong. This has to change.
It’s more like 4a. The line of thinking seems useful, but I’m not sure that it lands.
I appreciate the answer, but my concerns remain.
1. Doesn’t decompose manipulation, it just observes that manipulation will tend to be incentivized, so 2. we can do better by not allowing manipulation.
But for 2., how do we get an automated system and containment setup that is secure against a superintelligence? That’s where the difficulty seems to be transferred: we assume we have a machine immune to this “manipulation” thing, and then we can conclude there’s no manipulation. But is the global optimum for a stock-tracking system “correctly answering the query” instead of, like, “subtly modify the machine to always match this very output”, or “use the machine to escape anyways”? Humans can’t even check for this, because we assumed we won’t peek!
But that seems optimistic. We assume, contra to 4., that it has a model of the world and its future rewards, so what if it exploits some vulnerability to ensure it can manipulate us during the “innocent” step four? Again, we’ve already assumed we can’t check it during the erasure training, and if we check its code afterwards to make sure it’s no longer learning, that source could have manipulative content as well. (I’m generally confused about what capabilities are assumed—is it just souped-up modern ML? I anticipate that even if I misunderstand some points, I anticipate that there exists a steelmanned translation of this concern.)
Even worse, it could (if sufficiently intelligent) subtly transfer or otherwise preserve itself before being shut down. Why are we assuming we can just shut it down, given that we have to give it at least a little time to think and train? This isn’t acknowledged anywhere I’ve read. I’d be more comfortable if we had an oracle which doesn’t want out of the box, like one that wants to answer questions without becoming more able to answer questions. (That’s along the lines of an AUP oracle)
In my mind, there’s a notion of taking advantage of conceptual insights to make the superintelligent mountain less likely to be pushing against you. What part of a proposal is tackling a root cause of misalignment in the desired use cases? It’s alright if the proposal isn’t perfect, but heuristically I’d want to see something like “here’s an analysis of why manipulation happens, and here are principled reasons to think that this proposal averts some or all of the causes”.
Concretely, take CIRL, which I’m pretty sure most agree won’t work for the general case as formulated. In addition to the normal IRL component, there’s the insight of trying to formalize an agent cooperatively learning from a human. This contribution aimed to address a significant component of value learning failure.
(To be sure, I think the structure of “hey, what if the AI anticipates not being on anyways and is somehow rewarded only for accuracy” is a worthwhile suggestion and idea, and I am glad Stuart shared it. I just am not presently convinced it’s appropriate to conclude the design averts manipulation incentives / is safe at present.)
TurnTrout’s shortform feed
Thanks for clarifying! I haven’t brought this up on your research agenda because I prefer to have the discussion during an upcoming sequence of mine, and it felt unfair to comment on your agenda, “I disagree but I won’t elaborate right now”.
I also am curious why this should be so.
I also continue to disagree with Stuart on low impact in particular being intractable without learning human values.
I really like this layout, this idea, and the diagrams. Great work.
I don’t agree that counterfactual oracles fix the incentive. There are black boxes in that proposal, like “how is the automated system not vulnerable to manipulation” and “why do we think the system correctly formally measures the quantity in question?” (see more potential problems). I think relying only on this kind of engineering cleverness is generally dangerous, because it produces safety measures we don’t see how to break (and probably not safety measures that don’t break).
Also, on page 10 you write that during deployment, agents appear as if they are optimizing the training reward function. As evhub et al point out, this isn’t usually true: the objective recoverable from perfect IRL on a trained RL agent is often different (behavioral objective != training objective).
I’m confused what “corrigible alignment” means. Can you expand?
My main problem with these kinds of approaches is they seem to rely on winning a game of engineering cleverness against a superintelligent mountain of otherwise-dangerous optimization pressure. If we acknowledge that by default a full oracle search over consequences basically goes just as wrong as a full sovereign search over consequences, then the optimum of this agent’s search is only desirable if we nail the engineering and things work as expected. I have an intuition that this is highly unlikely—the odds just seem too high that we’ll forget some corner case (or even be able to see it).
ETA: I see I’ve been strongly downvoted, but I don’t see what’s objectionable.
If optimizing an arbitrary somewhat-but-not-perfectly-right utility function gives rise to serious Goodhart-related concerns
One thing I’ve been thinking about recently is: why does this happen? Could we have predicted the general phenomenon in advance, without imagining individual scenarios? What aspect of the structure of optimal goal pursuit in an environment reliably produces this result?
“AUP is not about state”—what does it mean for a method to be “about state”?
Here’s a potentially helpful analogy. Imagine I program a calculator. Although its computation is determined by the state of the solar system, the computation isn’t “about” the state of the solar system.
What do you mean, we can grab an evaluator? What I’m thinking of is similar to “IRL requires locating a human in the environment and formalizing their actions, which seems fuzzy”.
And if we can’t agree informally on deception’s definition, I’m saying “how can we say a proposal has the property”.
so I still don’t understand the details, so maybe my opinion will change if I sit down and look at it more carefully. But I’m suspicious of this being a clean incentive improvement that gets us what we want, because defining the evaluator is a fuzzy problem as I understand it, as is even informally agreeing on what counts as deception of a less capable evaluator. in general, it seems that if you don’t have the right formalism, you’re going to get Goodharting on incorrect conceptual contours.
Meta: I’d have appreciated a version with less math, because extra formalization can hide the contribution. Or, first explain colloquially why you believe X, and then show the math that shows X.
I don’t see your claim. It looks heavily incentivized to steer state sequences to be desirable to its utility mixture. How do the evaluators even enter the picture?
Hi. I’m also a PhD student (upcoming fourth year). Last year, I wrote about my process of auto didacting and switching research areas to directly work on FAI. Beyond that advice, I have more which is yet unpublished: select your committee wisely. I purposefully chose those professors who were easy to get along with and who seemed likely to be receptive to longer-term concerns.
I generally recommend working on an alignment research area in your free time, levelling up as necessary. One path I took: make some proposals and demonstrate you can think novel thoughts and do research. Then, get funding to make this your full-time research.
Alternatively, you can just level up while in your program. Check out Critch’s Deliberate Grad School and Leveraging Academia.
If you’re interested in a Skype sometime, feel free to message me. There’s a MIRIx Discord server I can invite you to as well.
Welcome to the journey!