I’m not totally sure I understood your comment, but I’ll try to say a few things that might be useful:
There’s a trivial sense in which Oversight-as-Inference is true, which is that I can make the input to the Python program be some type of empirically-backed explanation, and then create a judge that is “How convincing is this as an explanation of [thing X we care about], on a scale of 1-10?” I don’t think this literal judge works but I do think something along these lines works conceptually (modulo the immense computational cost of actually generating the program that makes the judge pass).
Re: generalization, I don’t generally think of this as trying to “reduce a generalization gap”; I kind of expected in-distribution generalization (plus the sort of “normal” OOD generalization we already see with LLM pretraining) to be enough.
I do think you probably need significant RL and other forms of training-time and test-time compute to get a system that is good enough to really move the needle on alignment. I expect that before that happens, we will get utility more along the lines of “100x cost reductions of things we could already do”. This seems somewhat analogous to how, for a while, AI assistants made math much cheaper but didn’t produce new math, but now they are producing new math. I think it’s almost directly analogous because math and oversight have the same verified rewards structure and are also both problems that humans are natively mediocre at.
Does that help? I’m not sure how to operationalize your objection in these terms—maybe it’s that you think the RL stage isn’t going to work?
I’m not totally sure I understood your comment, but I’ll try to say a few things that might be useful:
There’s a trivial sense in which Oversight-as-Inference is true, which is that I can make the input to the Python program be some type of empirically-backed explanation, and then create a judge that is “How convincing is this as an explanation of [thing X we care about], on a scale of 1-10?” I don’t think this literal judge works but I do think something along these lines works conceptually (modulo the immense computational cost of actually generating the program that makes the judge pass).
Re: generalization, I don’t generally think of this as trying to “reduce a generalization gap”; I kind of expected in-distribution generalization (plus the sort of “normal” OOD generalization we already see with LLM pretraining) to be enough.
I do think you probably need significant RL and other forms of training-time and test-time compute to get a system that is good enough to really move the needle on alignment. I expect that before that happens, we will get utility more along the lines of “100x cost reductions of things we could already do”. This seems somewhat analogous to how, for a while, AI assistants made math much cheaper but didn’t produce new math, but now they are producing new math. I think it’s almost directly analogous because math and oversight have the same verified rewards structure and are also both problems that humans are natively mediocre at.
Does that help? I’m not sure how to operationalize your objection in these terms—maybe it’s that you think the RL stage isn’t going to work?