I am very surprised that “Iterated Amplification” appears nowhere on this list. Am I missing something?
skluug
Karma: 17
More generally, I think that if mere-humans met very-alien minds with similarly-coherent preferences, and if the humans had the opportunity to magically fulfill certain alien preferences within some resource-budget, my guess is that the humans would have a pretty hard time offering power and wisdom in the right ways such that this overall went well for the aliens by their own lights (as extrapolated at the beginning), at least without some sort of volition-extrapolation.
Isn’t the worst case scenario just leaving the aliens alone? If I’m worried I’m going to fuck up some alien’s preferences, I’m just not going to give them any power or wisdom!
I guess you think we’re likely to fuck up the alien’s preferences by light of their reflection process, but not our reflection process. But this just recurs to the meta level. If I really do care about an alien’s preferences (as it feels like I do), why can’t I also care about their reflection process (which is just a meta preference)?
I feel like the meta level at which I no longer care about doing right by an alien is basically the meta level at which I stop caring about someone doing right by me. In fact, this is exactly how it seems mentally constructed: what I mean by “doing right by [person]” is “what that person would mean by ‘doing right by me’”. This seems like either something as simple as it naively looks, or sensitive to weird hyperparameters I’m not sure I care about anyway.
I feel like you say this because you expect your values-upon-reflection to be good by light of your present values—in which case, you’re not so much valuing reflection, as just enacting your current values.
If Omega told me if I reflected enough, I’d realize what I truly wanted was to club baby seals all day, I would take action to avoid ever reflecting that deeply!
It’s not so much that I want to lock in my present values as it is I don’t want to lock in my reflective values. They seem equally arbitrary to me.
This kind of thing seems totally backwards to me. In what sense do I lose if I “bulldoze my values”? It only makes sense to describe me as “having values” insofar as I don’t do things like bulldoze them! It seems like a way to pretend existential choices don’t exist—just assume you have a deep true utility function, and then do whatever maximizes it.
Why should I care about “teasing out” my deep values? I place no value on my unknown, latent values at present, and I see no reason to think I should!
Iterated Amplification is a fairly specific proposal for indefinitely scalable oversight, which doesn’t involve any human in the loop (if you start with a weak aligned AI). Recursive Reward Modeling is imagining (as I understand it) a human assisted by AIs to continuously do reward modeling; DeepMind’s original post about it lists “Iterated Amplification” as a separate research direction.
“Scalable Oversight”, as I understand it, refers to the research problem of how to provide a training signal to improve highly capable models. It’s the problem which IDA and RRM are both trying to solve. I think your summary of scalable oversight:
is inconsistent with how people in the industry use it. I think it’s generally meant to refer to the outer alignment problem, providing the right training objective. For example, here’s Anthropic’s “Measuring Progress on Scalable Oversight for LLMs” from 2022:
It references “Concrete Problems in AI Safety” from 2016, which frames the problem in a closely related way, as a kind of “semi-supervised reinforcement learning”. In either case, it’s clear what we’re talking about is providing a good signal to optimize for, not an AI doing mechanistic interpretability on the internals of another model. I thus think it belongs more under the “Control the thing” header.
I think your characterization of “Prosaic Alignment” suffers from related issues. Paul coined the term to refer to alignment techniques for prosaic AI, not techniques which are themselves prosaic. Since prosaic AI is what we’re presently worried about, any technique to align DNNs is prosaic AI alignment, by Paul’s definition.
My understanding is that AI labs, particularly Anthropic, are interested in moving from human-supervised techniques to AI-supervised techniques, as part of an overall agenda towards indefinitely scalable oversight via AI self-supervision. I don’t think Anthropic considers RLAIF an alignment endpoint itself.