The main thing which convinced me to start paying attention to corrigibility was: by that same argument, corrigibility is itself a part of human values. Which means that, insofar as some class of utility maximizers has trouble expressing corrigibility… that class will also have trouble expressing human values.
The way you phrase this is making me a bit skeptical. Just because something is part of human values doesn’t necessarily imply that if we can’t precisely specify that thing, it means we can’t point the AI at the human values at all. The intuition here would be that “human values” are themselves a specifically-formatted pointer to object-level goals, and that pointing an agent at this agent-specific “value”-type data structure (even one external to the AI) would be easier than pointing it at object-level goals directly. (DWIM being easier than hand-coding all moral philosophy.)
Which isn’t to say I buy that. My current standpoint is that “human values” are too much of a mess for the aforementioned argument to go through, and that manually coding-in something like corrigibility may be indeed easier.
Still, I’m nitpicking the exact form of the argument you’re presenting.[1]
Although I am currently skeptical even of corrigibility’s tractability. I think we’ll stand a better chance of just figuring out how to “sandbox” the AGI’s cognition such that it’s genuinely not trying to optimize over the channels by which it’s connected to the real world, then set it down the task of imagining the solution to alignment or to human brain uploading or whatever.
With this setup, if we screw up the task’s exact specification, it shouldn’t even risk exploding the world. And “doesn’t try to optimize over real-world output channels” sounds like a property for which we’ll actually be able to derive hard mathematical proofs, proofs that don’t route through tons of opaque-to-us environmental ambiguities. (Specifically, that’d probably require a mathematical specification of something like a Cartesian boundary.)
(This of course assumes us having white-box access to the AI’s world-model and cognition. Which we’ll also need here for understanding the solutions it derives without the AI translating them into humanese – since “translate into humanese” would by itself involve optimizing over the output channel.)
And it seems more doable than solving even the simplified corrigibility setup. At least, when I imagine hitting “run” on a supposedly-corrigible AI vs. a supposedly-sandboxed AI, the imaginary me in the latter scenario is somewhat less nervous.
The way you phrase this is making me a bit skeptical. Just because something is part of human values doesn’t necessarily imply that if we can’t precisely specify that thing, it means we can’t point the AI at the human values at all. The intuition here would be that “human values” are themselves a specifically-formatted pointer to object-level goals, and that pointing an agent at this agent-specific “value”-type data structure (even one external to the AI) would be easier than pointing it at object-level goals directly. (DWIM being easier than hand-coding all moral philosophy.)
Which isn’t to say I buy that. My current standpoint is that “human values” are too much of a mess for the aforementioned argument to go through, and that manually coding-in something like corrigibility may be indeed easier.
Still, I’m nitpicking the exact form of the argument you’re presenting.[1]
Although I am currently skeptical even of corrigibility’s tractability. I think we’ll stand a better chance of just figuring out how to “sandbox” the AGI’s cognition such that it’s genuinely not trying to optimize over the channels by which it’s connected to the real world, then set it down the task of imagining the solution to alignment or to human brain uploading or whatever.
With this setup, if we screw up the task’s exact specification, it shouldn’t even risk exploding the world. And “doesn’t try to optimize over real-world output channels” sounds like a property for which we’ll actually be able to derive hard mathematical proofs, proofs that don’t route through tons of opaque-to-us environmental ambiguities. (Specifically, that’d probably require a mathematical specification of something like a Cartesian boundary.)
(This of course assumes us having white-box access to the AI’s world-model and cognition. Which we’ll also need here for understanding the solutions it derives without the AI translating them into humanese – since “translate into humanese” would by itself involve optimizing over the output channel.)
And it seems more doable than solving even the simplified corrigibility setup. At least, when I imagine hitting “run” on a supposedly-corrigible AI vs. a supposedly-sandboxed AI, the imaginary me in the latter scenario is somewhat less nervous.