While these proofs do show that honesty remains incentivized in the superhuman limit, both the protocols and their proofs are difficult to apply to real-world problems.
This seems misleading—both on “honesty” and “incentivized”.
It’s not about “difficult to apply”—it’s about providing neither the kind of honesty we’d need, nor the kind of incentive we’d need, without reliance on unsound assumptions.
A place where you seem to be straying into similar territory (see the “honesty” section below) is:
Let’s define ground truth to be what the judge would conclude given access to all possible arguments, i.e. the joint distribution for the entire factor graph.
Why???
By all means define a label signifying [what the judge would conclude given access to all possible arguments]. Even assuming this were well-defined (e.g. not order-dependent), why would we want to call it ground truth?
For some judges it’s extremely obviously not the ground truth; for most others it’s more-or-less-obviously not. The distinction matters hugely in a safety context.
I expect there are a bunch of interesting results one could show around approximation of, or convergence to [what the judge would conclude...]. I think it’s highly unhelpful to call that “ground truth” simply because it’s the closest available approximation in the context.
On the “honesty” part (in the original quote):
First, prover-estimator debate is all relative to some “human judgement oracle”. So it wouldn’t be showing “honesty remains incentivized”—rather something like “honesty, defined relative to some oracle, remains incentivized”.
Second, human judgement is systematically biased on essentially every question (there’s no reason to expect precise symmetry of upward and downward error in probability estimates, even in simple cases). It’s also inconsistent.
So a self-consistent oracle isn’t a useful idealization of human judgement—it assumes away core difficulties. (as soon as we drop a self-consistency assumption, we’d need to adopt some kind of reflective self-‘correction’ mechanism, and halting condition—and judge self-‘correction’ under extreme optimization pressure doesn’t seem likely to end well by default)
It’s worth noting that honesty has little value if a system has been trained to believe [whatever set of beliefs best exploit the biases and inaccuracies of humans]. Nor is this a benign failure, since such a system would appear to be making true claims based on sound arguments.
In the safety case paper, the equivalent is defining honesty relative to “a process we hypothesize to produce true outputs, called M”.
M grounds out in human judgement—which is systematically biased on essentially every question, and not self-consistent. So the hypothesis that M produces true outputs is pretty dubious ([M produces outputs favored by optimal exploitation of human bias] seems rather more plausible).
So the thing that remains “incentivized” is not honesty-in-the-sense-we’d-want. (it’s something like [faithfulness to a maximally-persuasive-to-a-human-judge set of beliefs] - i.e. a kind of deep sycophancy)
On “incentivized”:
For a safety guarantee, we’d need to know that systems selected by a training process would have some desirable property.
What matters are the equilibria of the training process, not the equilibria of the game we use to generate the training signal. We’d need a strong convergence assumption even to begin to argue that the latter dictates the former (something along the lines of a guarantee that the most effective honest strategies are actually explored).
Such an exploration assumption still wouldn’t give us any kind of generalization guarantee: it’s consistent with [system debates honestly until it sees some opportunity to defect]. (defecting might mean that the system loses the debate game; but it need not care about that (and if there’s collusion with the other debater, it might not even lose: defection may go entirely unnoticed))
The incentive is only ever conditional:
- if we assume that debate systems aim purely to win the debate game, and do by sticking to the protocol (e.g. no side-channel attacks...), we get to make claims about ‘honest’ behavior.
But we can’t make this assumption, for standard inner alignment reasons.
Nor does it seem at all reasonable to stipulate that a natural training process will happen to bump into a [perfectly honest, purely and deeply motivated to win at debate] system during training, such that we might start to argue that honesty will win from that point onward (and perhaps that robust generalization guarantees will come along for the ride, because… ???).
I’m not sure what to think about this comment (or the disagreement votes), since you seem to be comparing two approaches I’d expect to fail:
Training “alignment taste” into models, so they can do alignment research.
To a first approximation, this seems to relabel “alignment” as “alignment taste” so that we can pretend there’s no infinite regress. We don’t need [model in some sense knows the right information]; we need [model robustly gives us the right information]. When I think about the required sense of “alignment taste”, it looks suspiciously like alignment.
Not in the sense of being logically inseparable.
In the sense of being a similarly hard problem.
(likewise for attempts that route through a motte-and-bailey on [not egregiously misaligned]).
Hillclimbable tasks:
This seems like a natural way to address [the subset of issues we can clearly see]. (and to expand that set to an extent)
The standard outcome here is to widen the gap between perceived accuracy and actual accuracy (or [perceived x] and [actual x] for some desirable x).
I can imagine many good arguments for [but using y we can make this effect less extreme than it otherwise would be].
Sure—but that’s not the bar.
We’ll predictably reach progressively higher and more general versions of “Well, it sure looks good to us...”.
What’s the process to estimate p(actually good | looks good)??
This is not a rhetorical question!
The usual answer I expect here has the form:
The thing we’re doing seems clearly useful, but coming up with a principled process to estimate p(actually good | looks good) for some overall approach is not our department.
In which case, I’d like to know where that department is, and why I ought to expect that department to do a surprisingly-to-me good job.
More generally, I’d appreciate any pointer to an argument of the form [we should expect alignment automation to work (or that we’re on the path to this)] rather than [this approach to alignment automation would be less obviously bad than various other approaches].
I’m aware of:
(my views provided for context; no need to respond to them!)
How might we safely pass the buck to AI? (Clymer) (which just looks like wishful thinking)
Related: Training AI to do alignment research we don’t already know how to do.
(more wishful thinking and much motte-and-bailey)
In particular, there’s an explicit “What matters is that this process yields better conclusions than humans would otherwise arrive at”, which is just false without unrealistic assumptions around overconfidence in strategy.
For example, if the AI approach gives us a 5% chance of avoiding disaster, to the humans’ 1%, this is only an improvement if the AI approach is mistrusted to a similar level as the human approach.
If the AI is shaping strategy too, then the reasonable comparison is to [what we ought to do], rather than [what foolish people will do by default].
“We ought to be pretty foolish, because by default we’ll be extremely foolish” is not a good argument. If the default is terrible, then [better than the default] is clearly not the bar.
The obviously sensible high-level strategy is to take the hugely more indirect approach that routes through deep understanding (using AI in some fashion).
Naturally we “can” use AI this way—but this is the wrong question.
Can we safely automate alignment research? (Carlsmith)
Issues with this include:
It’s the wrong question: we need to know that there’s a reasonable chance that we actually will safely automate alignment research with some particular process. (this isn’t a quibble over words: the content argues for “can”, which is almost empty)
Focusing on the wrong problem—see this Capability vs Safety graph.
The core issue is not [what’s our capability level vs safety level?].
Rather it’s [what’s our perceived safety level vs actual safety level?].
And the fact that it’s high-dimensional makes this much worse—it’s not enough to be conservative in almost every necessary sense.
(again, not a quibble over graph choice—rather a critique of the form and focus of the entire series)
Yes, Carlsmith’s model can be considered to include this under Risk Evaluation, but I think that this is actively misleading. Risk evaluation is the problem.
Various arguments for scalable oversight that have the form:
We can use this to do a deep kind of reflectively self-improving process supervision.
If that doesn’t work, then we have no idea what will… so let’s hope that it works.
(again, this is an [x seems better than other ys we can think of] argument, not an [x is actually likely to work] argument)
Are there any better high-level cases?