Yet another example is @abramdemski’s “Vingean agency” (2022) (following earlier work by Yudkowsky). He starts from a place very close to the human intuitions in §2 above, i.e. “agency” is when you can predict the outcome but not the actions leading to those outcomes. Then he hints at an intriguing idea that maybe we can just make that formal! I.e., maybe I was too quick to dismiss that kind of thing above using terms like “messed-up ontology”. (As Abram writes: “I also think it’s possible that Vingean agency can be extended to be ‘the’ definition of agency, if we think that agency is just Vingean agency from some perspective….”)
By analogy, to borrow an example from @johnswentworth, thermodynamics concepts like “temperature” are tied to imperfect modeling ability (since an omniscient observer would instead track the velocity of every particle). So why can’t “agency” be tied to imperfect modeling ability too?
But alas, even if we can rigorously define Vingean agency, I don’t think it would really help with the problem I want it to solve here, i.e. pinning down a distinction between good “counsel” vs bad “manipulation”. Vingean agency seems to solve the problem of identifying an agent trying to do something, by noticing easier-to-predict ends happening by harder-to-predict means. But the “manipulation” concept worries about the possibility of intervention upstream of a person’s ego-syntonic desires. If the AI can brainwash me into deeply wanting to maximize paperclips, and then I execute a clever plan to maximize paperclips, then I would still be a Vingean agent, as long as my clever plan was sufficiently clever (from some perspective). So the brainwashing would strip me of my intuitive agency, but not my Vingean agency.
I don’t think I’ve done enough work to boldly state that I can solve your problem, but I do think you seem overly pessimistic here. The most current attempt to communicate the idea is in my post Legitimate Deliberation.
Roughly, Vingean Agency can be identified via modified variants of van Fraassen’s reflection principle (roughly: I don’t know what move the chess grandmaster will make, but I do know that if I were trying to win a so-far identical game, I’d copy the grandmaster’s moves given the option). This only ever makes sense from a computationally bounded perspective; agents (conceived thusly) vanish in the limit of cognition.
van Fraassen invented this principle not to study the epistemic state of a novice thinking about a grandmaster, but rather, to study our own regard for our future opinions. Our preferences can change, but (ideally) we consider those changes to be correct; not in the sense that our new opinions must be correct, nor even in the sense that our new opinions are necessarily revised in the correct direction; but in the sense of van Fraassen’s reflection principle.
The literature was quick to point out that there are exceptions to the principle: we expect to forget many details of the day, such as what we had for lunch, how many times we went to the bathroom, etc as time goes on. We can expect our future opinions to be less accurate rather than more for such examples. Other examples include getting drunk, being gaslit, etc.
I use the terms “legitimate” vs “illegitimate” to describe opinion-revision processes that do and do not respect van Fraassen’s reflection principle. This is just a definition of convenient terminology, not a deep account; what makes something legitimate vs illegitimate is still a big question. I expect the answer is complex in the same way that human value is complex.
Still, I think “legitimacy” provides a better handle on what the AI is supposed to be avoiding than “manipulation”—many humans do not want to experience heroin, even though they’d predictably enjoy it and want more, because they do not consider that change a legitimate one. Similarly, humans would not want to talk to manipulative AI.
I have recently been thinking about how to construct ML training based on this design goal. Suppose you have a slow but trustworthy belief-revision process, and a fast, highly capable but untrusted belief-revision process—which is trustworthy on problems where it gets good feedback. It seems potentially possible to create a fast-and-trustworthy process by asking the fast process to predict the slow process. (This fits the basic picture of alignment as uploading with more steps). One advantage of this approach is that we don’t need to be able to give gold-standard feedback on any object-level questions; the AI is instead trained on our opinions and how they shift over time. The result is supposed to avoid human manipulation for the same reasons that (some) humans avoid hard drugs: because manipulation would violate the legitimacy of the feedback process. It is corrigible in the (weak) sense that, so long as it expects human feedback to be legitimate, cutting that feedback off would be negative-ev (viewed as lost information). Either it has already anticipated the belief change and updated accordingly (so there’s no reason to block modifications the humans want to implement) or it wants the information and so won’t block the update (because it trusts the humans) or it deems the human feedback illegitimate (which can either be because it is, or can be a mistake; in either case we’ve messed up the training process).
Thanks! Seems like the upshot is that humans have a complex soup of desires around learning from and epistemically interacting with AIs, and you’re proposing to use the word “legitimate” to describe that soup, which I agree is a helpful way of thinking about it (and probably better than “not-manipulating”), but also not a True Name in the John Wentworth sense.
The proposal in the last paragraph seems unhelpful for me-in-particular, because I’m working on how to install goals in brain-like AGIs, and I have ideas that seem promising but only work for a limited number of goals (they kinda have to be simple, concrete, “atomic”, and/or directly related to people’s feelings, and/or have a ground truth that can be calculated explicitly, more-or-less). The last paragraph seems to entail making an AGI that wants humans to come quickly to the conclusions that they would otherwise have reached eventually on their own. But that’s pretty complex and abstract, and not something I have any good idea of how to install as a goal / motivation, alas.
No comment about whether it’s a viable approach in a more LLM-like paradigm.
I don’t think I’ve done enough work to boldly state that I can solve your problem, but I do think you seem overly pessimistic here. The most current attempt to communicate the idea is in my post Legitimate Deliberation.
Roughly, Vingean Agency can be identified via modified variants of van Fraassen’s reflection principle (roughly: I don’t know what move the chess grandmaster will make, but I do know that if I were trying to win a so-far identical game, I’d copy the grandmaster’s moves given the option). This only ever makes sense from a computationally bounded perspective; agents (conceived thusly) vanish in the limit of cognition.
van Fraassen invented this principle not to study the epistemic state of a novice thinking about a grandmaster, but rather, to study our own regard for our future opinions. Our preferences can change, but (ideally) we consider those changes to be correct; not in the sense that our new opinions must be correct, nor even in the sense that our new opinions are necessarily revised in the correct direction; but in the sense of van Fraassen’s reflection principle.
The literature was quick to point out that there are exceptions to the principle: we expect to forget many details of the day, such as what we had for lunch, how many times we went to the bathroom, etc as time goes on. We can expect our future opinions to be less accurate rather than more for such examples. Other examples include getting drunk, being gaslit, etc.
I use the terms “legitimate” vs “illegitimate” to describe opinion-revision processes that do and do not respect van Fraassen’s reflection principle. This is just a definition of convenient terminology, not a deep account; what makes something legitimate vs illegitimate is still a big question. I expect the answer is complex in the same way that human value is complex.
Still, I think “legitimacy” provides a better handle on what the AI is supposed to be avoiding than “manipulation”—many humans do not want to experience heroin, even though they’d predictably enjoy it and want more, because they do not consider that change a legitimate one. Similarly, humans would not want to talk to manipulative AI.
I have recently been thinking about how to construct ML training based on this design goal. Suppose you have a slow but trustworthy belief-revision process, and a fast, highly capable but untrusted belief-revision process—which is trustworthy on problems where it gets good feedback. It seems potentially possible to create a fast-and-trustworthy process by asking the fast process to predict the slow process. (This fits the basic picture of alignment as uploading with more steps). One advantage of this approach is that we don’t need to be able to give gold-standard feedback on any object-level questions; the AI is instead trained on our opinions and how they shift over time. The result is supposed to avoid human manipulation for the same reasons that (some) humans avoid hard drugs: because manipulation would violate the legitimacy of the feedback process. It is corrigible in the (weak) sense that, so long as it expects human feedback to be legitimate, cutting that feedback off would be negative-ev (viewed as lost information). Either it has already anticipated the belief change and updated accordingly (so there’s no reason to block modifications the humans want to implement) or it wants the information and so won’t block the update (because it trusts the humans) or it deems the human feedback illegitimate (which can either be because it is, or can be a mistake; in either case we’ve messed up the training process).
Thanks! Seems like the upshot is that humans have a complex soup of desires around learning from and epistemically interacting with AIs, and you’re proposing to use the word “legitimate” to describe that soup, which I agree is a helpful way of thinking about it (and probably better than “not-manipulating”), but also not a True Name in the John Wentworth sense.
The proposal in the last paragraph seems unhelpful for me-in-particular, because I’m working on how to install goals in brain-like AGIs, and I have ideas that seem promising but only work for a limited number of goals (they kinda have to be simple, concrete, “atomic”, and/or directly related to people’s feelings, and/or have a ground truth that can be calculated explicitly, more-or-less). The last paragraph seems to entail making an AGI that wants humans to come quickly to the conclusions that they would otherwise have reached eventually on their own. But that’s pretty complex and abstract, and not something I have any good idea of how to install as a goal / motivation, alas.
No comment about whether it’s a viable approach in a more LLM-like paradigm.