I’m kind of bouncing off the “corrigibility” terminology used in this post. It doesn’t seem to me very conceptually complex to build an agent (in the AI sense) that knows “I’m an agent (in the Economics sense), and my goal is to look out for my principal’s interests, and part of the challenge of doing this is that I don’t yet fully know what those are”. That leads directly to a number of behaviors: a) asking the principal for more information, b) Bayesian updating when the principal offers corrections (behavior which I would use the term “corrigibility” for), c) studying evidence about the principal’s preferences and interests, d) making predictions from scientific knowledge such as Evolutionary Psychology about the principal. I’d want an AGI/ASI to do all of these. As it became an ASI, I’d expect it to increasingly do c) and d) and increasingly correct itself rather than us needing to, and perhaps to gradually rely less on b) (at least unless it received a correction that actually surprised it). So I don’t like using the term “Corrigibility” for this entire motivation and resulting behavior set, as that’s naming it by just one of several consequences. Which is why I prefer the older term “Value Learning”, or the newer term “AI-Assisted Alignment”, which are what I used when I wrote Alignment has a Basin of Attraction: Beyond the Orthogonality Thesis
I also think that any sufficiently-capable and sufficiently-close-to-aligned ASI will figure this out and start to do this, whether you engineer it in or not (though I’d definitely advocate doing so deliberately). A sufficiently-close to aligned agent already knows and believes “I’m an agent (in the Economics sense), and my goal is to look out for my principal’s interests”, so it only has to notice the second half of the sentence — which is really rather obvious to anything intelligent that isn’t actually built like AIXI.
That’s also why I’m less concerned we’re going to “run out of fuel” as we get closer to ASI — that’s when the AI can do more and more of the work itself. Which increasingly isn’t “corrigibility”, it’s more behaviors c) and d) above, but that’s fine, as long as the agent is capable enough to do it right. And if it isn’t, then we fall back on b), actual corrigibility.
The more interesting question to me is whether an LLM is a good place to start to achieve this. In one sense, it doesn’t seem that bad: you can just hand an LLM a system prompt that says that’s the persona, please play that. Fundamentally, it’s expressible in one sentence (though I’d suggest giving more background). However, you’re asking for a persona which, while easy to define, is extremely rare in the human-derived training set that base models are trained on. You might need to supplement that with hand-written or synthetic data about how such an agent would act.
I’m going to ask my implicit question again, explicitly this time:
At four points spread through your post you write things functionally equivalent to:
”we will run out of fuel before resolving the main problems”
I have read your post twice, carefully, and I still have no idea what “fuel” is intended to be a metaphor for here, or why you assert that “we’re going to run out of it”. I genuinely don’t know what argument you’re making. I’d like to understand your argument, so that I can consider and engage with it.
I do have a number of hypotheses (which may all be wrong — if so, please tell me):
You’re gesturing at (without mentioning) the old problem of “fully-updated deference”: that if we assume that our AI is an excellent Bayesian, and we have no other way to make it corrigible apart from its Bayesian uncertainty about the human values we’re aligning it to, then we can’t keep handing it contradictory corrections indefinitely past the point where this degree of changing our minds about what we want becomes implausible to a good Bayesian, and it decides to stop paying very much attention to us because we’re clearly a random number generator rather than a source of information about our own values — that the AI (by assumption, an excellent Bayesian) will run out of Bayesian uncertainty before the problem is solved (which by definition can only happen if we waste an implausible amount of it).
You’re talking about the fact (true in absolutely every field of science and engineering, especially safety engineering fields) that extrapolating out of your current distribution is hard work and requires you to proceed slowly, carefully and painstakingly, expanding the distribution a little bit at a time (such as by, at some point once you are sufficiently confident from lake-sailing the ship, taking it out to sea for half and hour on an extremely calm and waveless day, and then returning to shore to analyze all your sensor readings for a long time) — that we’ll run out of tests we can do reasonably safely, and their outcomes will not teach us enough to enable us to come up with any more tests that we are now able to do reasonably safely.
a. You’re concerned that if we do enough tests that we believe to be reasonably safe, one of them that isn’t done on the lake but actually at sea will fail so badly that we all die — that we’ll run out of luck. Or perhaps: b. You’re thus absolutely unwilling to do any such tests, even during the ship’s official maiden voyage and next 10 years at sea — which seems like a wasted opportunity, if the ship’s officers are taking it to sea anyway with us on board.
You feel that maritime safety engineering, and presumably by analogy AI safety engineering, sounds dull and detail-oriented and painstaking, and you think people will get bored and give up rather than solving all the problems involved carefully and iteratively (even when existential stakes are involved) — that we’ll run out of enthusiasm for the work.
Something else that I haven’t thought of.
Some or all of the above.
(You might also wish to consider reediting your post, or even writing a new one, in case I’m not the only person confused what your argument is.)
2 is close enough. Extrapolating the results of safe tests to unsafe settings requires a level of theoretical competence we don’t currently have. Steve Brynes just made a great post that is somewhat related, I endorse everything in that post.
Thanks. I’ll reread you post again in light of that. I guess I was always assuming that doing that was going to become necessary, as it is in every field, and as is particularly challenging to do safely in safety engineering fields. Also that we were currently only, say O(10%) of the way through the process. So I’m unsurprised we can’t see much of the route yet — but we can see more of it than we could, say, 5 years ago, and usually we do figure these things out eventually. What concerns me isn’t that this is impossible, but that I don’t think we’re on track to be done in another, say, 5 years, that rushing is is a great way to cause 3.a., and it’s unclear how long we have left.
I’ll also take a look at Steve Bryrnes post you link.
It doesn’t seem to me very conceptually complex to build an agent (in the AI sense) that knows “I’m an agent (in the Economics sense), and my goal is to look out for my principal’s interests, and part of the challenge of doing this is that I don’t yet fully know what those are”.
I’m not claiming it’s conceptually complex. The conceptual simplicity is a large part of the motivation for pursuing this. Some of the difficulty comes up when you try to design the implementation details and need it to also be reflectively stable and generalise appropriately. The other difficulty comes from different implementations being difficult to distinguish from a training perspective, but having different reflective stability and generalisation properties.
b) Bayesian updating when the principal offers corrections (behavior which I would use the term “corrigibility” for)
That was not ever the intended meaning of the word. Using it like that is a great way to confuse everyone.
Which is why I prefer the older term “Value Learning”, or the newer term “AI-Assisted Alignment”
Both of these terms have different meanings from what I intend. AI-assisted alignment is closely related, but it is a process made possible by stable correctability, not the same thing as stable correctablility. Having a term for stable correctability itself is useful.
A sufficiently-close to aligned agent already knows and believes “I’m an agent (in the Economics sense), and my goal is to look out for my principal’s interests”, so it only has to notice the second half of the sentence — which is really rather obvious to anything intelligent that isn’t actually built like AIXI.
I’m aware that there are certain kinds of almost-aligned that will correct themselves. But I don’t see this as very relevant, that kinds of brokenness was never the important kind, the important kind of design-error are the ones that affect a reflectively consistent degree of freedom.
That’s also why I’m less concerned we’re going to “run out of fuel” as we get closer to ASI — that’s when the AI can do more and more of the work itself.
You misunderstand my intended thesis, it’s not related to quantity of work, it’s closer to how much “ability to course-correct” that humans have.
> b) Bayesian updating when the principal offers corrections (behavior which I would use the term > “corrigibility” for)
That was not ever the intended meaning of the word. Using it like that is a great way to confuse everyone.
Let me make sure I’ve understood you. The AI does something, mistakenly thinking that it’ll make me happy. I am not happy, and tell it “Hey! Never do that again!”. The AI Bayesian updates its model of what humans like in light of this new data. How is that not correctly described as “corrigibility” (the ability to be corrected)? I corrected the agent, it incorporated this data appropriately into its model of how the world and humans work (including properly allowing for possibilities like that I was confused, drunk, or actually upset about something else, rather than simply taking my word for it because I’m human). That seems like, not just corrigibility, but exactly the ideal level of corrigibility. Are you using corrigibility in some sense other than “the humans have the capability to correct the AI’s mistaken beliefs about what its utility function/goals should be”? If so, perhaps you should define it?
Or are you concerned that the AI will run out of belief that it could be mistaken, or at least could have more to learn, too early: its posteriors will all irreversibly reach 99.9999999…% with no possibility that it needs to consider any previous unconsidered hypotheses about anything about human values or how best to optimized them before its beliefs about human values are actually functionally complete — i.e. that it’ll be “a bad Bayesian”? (Personally I tend to assume that bad Bayesians are not actually going to be able to take over the world and go Foom, but YMMV.)
You misunderstand my intended thesis, it’s not related to quantity of work, it’s closer to how much “ability to course-correct” that humans have.
I must admit, I did find your post hard to penetrate. The pessimism came across, the “but we can only test in toy situations” analogy about ships — but it wasn’t clear to me what you meant by “this will run out of fuel”, and I didn’t get why you appear to be assuming the AI can’t help, in a post about “a basin of attraction”. IMO, for us to be in a basin of attraction, we need to already have nearly-aligned, significantly-capable AI, that wants to help with alignment, can help with alignment, and actually will (mostly) help with alignment. And the requirement for convergence is that, net between us plus the AI, we continue on average to make forward progress, so the process converges rather than diverging. That doesn’t have to run out of fuel because we reached the edge of what humans could do unaided — but it does require that then the aid we’re getting is sufficiently trustworthy (at least after we’ve examined it carefully) to cause convergence rather than divergence.
That’s corrigible behaviour, but the mechanism is not, because it stops being corrigible after some number of updates. The idea of corrigibility is that it is correctable, and remains so, in spite of designers making mistakes in goal alignment (or other design mistakes). (Of course mistakes in however we enforced the corrigibility property itself might not be stably correctable, but the hope is that this part is easier to get right on the first try than the rest of it).
Often asserted, but simply untrue. Bayesian updates never stop being corrigible. If the sun stops rising in the East, people update. After a few failed sunrises, they update hard. Bayesian posteriors have the Martingale property: their future direction of change is not predictable from their current value (if it were, you should already have updated in that direction). So even if it’s very high, it’s not just possible but probable (a-priori, has a 50% chance) that the next update will be a drop. (For approximate Bayesian reasoners, this remains true, unless you have access to significantly more computational capacity than it does — a smarter agent may see something it missed.) It takes a mountain of evidence to drive a Bayesian posterior very high, but an equally large mountain of opposing evidence will always drive it right back down again. Or, more often, even a small hill of opposing evidence will cause a good approximate Bayesian to spawn a new hypothesis that it hadn’t previously considered, one more compatible with both the mountain and the hill of evidence than “two huge/large opposing coincidences occurred”. E.g. a vast amount of evidence supporting Newtonian Mechanics doesn’t disprove Relativity, if it’s all from situations where they give indistinguishable results. In general, if I give an approximate Bayesian reasoner even ~30-bits-worth of opposing evidence, it should start looking for new hypotheses rather than just assuming that it’s right and a one-in-a-billion coincidence has just occurred.
[If it doesn’t do this, it’s a worse Bayesian than humans are, and thus hopefully not that dangerous — if a conflict occurred, we could outsmart it.]
I’m familiar with the argument. I just don’t agree with it. I think fully updated deference is asking for the impossible: you want a rational agent to continue letting you change your mind (and its) indefinitely, and not ever come to the obvious conclusion that you aren’t telling it the truth or are in fact confused. Personally, I’m willing to accept that a human-or-better approximate Bayesian reasoner rationally deducing what human values are eventually will do at least as good a job as we can by correcting it, and will be aware of this, and thus will eventually stop giving us deference beyond simply treating us as a source of data points, other than out of politeness. So (to paraphrase a famous detective), having eliminated the impossible, whatever is left I am willing to term “corrigibility”. If your definition of “corrigibility” includes fully updated deference, then yes, I agree, it’s impossible to achieve on the basis of Bayesian uncertainty: the Bayesian will eventually realize you’re being unreasonable, if you make enough unreasonable demands, and stop listening to you. However, if you only correct it with good reason, and it’s a good Bayesian, then you won’t run out of corrigibility.
In short, I’m unwilling to accept redefining the everyday term “corrigibility” to include “something logically impossible”, and then saying that they’ve proven that corrigibility is impossible — it’s linguistic slight of hand. I would suggest instead creating a more accurate term, such as saying something like that “unreasonably-unlimited corrigibility” isn’t possible on the basis of Bayesian uncertainty. Which is, well, unsurprising.
Returning to your concern that we may “run out of fuel” — only if we waste it by making unreasonable demands. We have all the corrigibility we could actually need — a good Bayesian isn’t going to decide we’re untrustworthy and stop paying us deference unless we actually do something clearly untrustworthy, like expect the right to keep changing our mind indefinitely.
Also, this does mean “reasonable”: if society changed and the AI just went out of distribution, and is wrong as a result, we get to tell it so, and it should keep spawning new hypotheses and collecting new evidence until it’s fully updated in this distribution region as well — thus my discussion above of Relativity.
This also generalizes, in ways that I think do actually give you something pretty close to fully updated deference. Just as if the sun stopped rising in the East, people would update, if you give a “fully updated” Bayesian reasoner ~30 bits of evidence that you want to shut it down (for reasons that actually look like it’s made a mistake and you’re legitimately scared and not confused, rather than some obvious other human motive), it should say: either a vastly improbably one-in-a-billion event has just occurred, or there’s a hypothesis missing from my hypothesis space. The latter seems more plausible. Maybe what humans value just changed, or there’s something I’ve been missing so far? Let’s spawn some new hypotheses, consistent with both all the old data and this new “ought to be incredible improbable” observation. It sure looks like they’re really scared. I talked to them about this, and they don’t appear to simply be confused… (And if it doesn’t say that, give it another ~30 bits of evidence.)
I see that you’re doing large edits and additions to your previous responses, after I had already responded.
This, and the way you’re playing with definitions, makes me think you might be arguing in bad faith. I’m going to stop responding. If you had good intentions, I’m sorry.
I did have good intentions, I just like to make my exposition as clear and well-thought-through as possible. But that’s fine, I think we have rather different views, and you’re under no obligation to engage with mine. Your alternative would be to wait until my reply stabilizes before replying to it, which is generally O(an hour). Remaining typo density is another cue. Sadly there is no draft option on the replies, and I can’t be bothered to do the editing somewhere else. Most interloctors don’t reply as quickly as you have been, so this hasn’t previously caused problems that I’m aware of.
On “playing with definitions” — actually, I’m saying that, IMO, some thinkers associated with MIRI have done so (see the second paragraph of my previous post).
I’m kind of bouncing off the “corrigibility” terminology used in this post. It doesn’t seem to me very conceptually complex to build an agent (in the AI sense) that knows “I’m an agent (in the Economics sense), and my goal is to look out for my principal’s interests, and part of the challenge of doing this is that I don’t yet fully know what those are”. That leads directly to a number of behaviors: a) asking the principal for more information, b) Bayesian updating when the principal offers corrections (behavior which I would use the term “corrigibility” for), c) studying evidence about the principal’s preferences and interests, d) making predictions from scientific knowledge such as Evolutionary Psychology about the principal. I’d want an AGI/ASI to do all of these. As it became an ASI, I’d expect it to increasingly do c) and d) and increasingly correct itself rather than us needing to, and perhaps to gradually rely less on b) (at least unless it received a correction that actually surprised it). So I don’t like using the term “Corrigibility” for this entire motivation and resulting behavior set, as that’s naming it by just one of several consequences. Which is why I prefer the older term “Value Learning”, or the newer term “AI-Assisted Alignment”, which are what I used when I wrote Alignment has a Basin of Attraction: Beyond the Orthogonality Thesis
I also think that any sufficiently-capable and sufficiently-close-to-aligned ASI will figure this out and start to do this, whether you engineer it in or not (though I’d definitely advocate doing so deliberately). A sufficiently-close to aligned agent already knows and believes “I’m an agent (in the Economics sense), and my goal is to look out for my principal’s interests”, so it only has to notice the second half of the sentence — which is really rather obvious to anything intelligent that isn’t actually built like AIXI.
That’s also why I’m less concerned we’re going to “run out of fuel” as we get closer to ASI — that’s when the AI can do more and more of the work itself. Which increasingly isn’t “corrigibility”, it’s more behaviors c) and d) above, but that’s fine, as long as the agent is capable enough to do it right. And if it isn’t, then we fall back on b), actual corrigibility.
The more interesting question to me is whether an LLM is a good place to start to achieve this. In one sense, it doesn’t seem that bad: you can just hand an LLM a system prompt that says that’s the persona, please play that. Fundamentally, it’s expressible in one sentence (though I’d suggest giving more background). However, you’re asking for a persona which, while easy to define, is extremely rare in the human-derived training set that base models are trained on. You might need to supplement that with hand-written or synthetic data about how such an agent would act.
I’m going to ask my implicit question again, explicitly this time:
At four points spread through your post you write things functionally equivalent to:
”we will run out of fuel before resolving the main problems”
I have read your post twice, carefully, and I still have no idea what “fuel” is intended to be a metaphor for here, or why you assert that “we’re going to run out of it”. I genuinely don’t know what argument you’re making. I’d like to understand your argument, so that I can consider and engage with it.
I do have a number of hypotheses (which may all be wrong — if so, please tell me):
You’re gesturing at (without mentioning) the old problem of “fully-updated deference”: that if we assume that our AI is an excellent Bayesian, and we have no other way to make it corrigible apart from its Bayesian uncertainty about the human values we’re aligning it to, then we can’t keep handing it contradictory corrections indefinitely past the point where this degree of changing our minds about what we want becomes implausible to a good Bayesian, and it decides to stop paying very much attention to us because we’re clearly a random number generator rather than a source of information about our own values — that the AI (by assumption, an excellent Bayesian) will run out of Bayesian uncertainty before the problem is solved (which by definition can only happen if we waste an implausible amount of it).
You’re talking about the fact (true in absolutely every field of science and engineering, especially safety engineering fields) that extrapolating out of your current distribution is hard work and requires you to proceed slowly, carefully and painstakingly, expanding the distribution a little bit at a time (such as by, at some point once you are sufficiently confident from lake-sailing the ship, taking it out to sea for half and hour on an extremely calm and waveless day, and then returning to shore to analyze all your sensor readings for a long time) — that we’ll run out of tests we can do reasonably safely, and their outcomes will not teach us enough to enable us to come up with any more tests that we are now able to do reasonably safely.
a. You’re concerned that if we do enough tests that we believe to be reasonably safe, one of them that isn’t done on the lake but actually at sea will fail so badly that we all die — that we’ll run out of luck.
Or perhaps:
b. You’re thus absolutely unwilling to do any such tests, even during the ship’s official maiden voyage and next 10 years at sea — which seems like a wasted opportunity, if the ship’s officers are taking it to sea anyway with us on board.
You feel that maritime safety engineering, and presumably by analogy AI safety engineering, sounds dull and detail-oriented and painstaking, and you think people will get bored and give up rather than solving all the problems involved carefully and iteratively (even when existential stakes are involved) — that we’ll run out of enthusiasm for the work.
Something else that I haven’t thought of.
Some or all of the above.
(You might also wish to consider reediting your post, or even writing a new one, in case I’m not the only person confused what your argument is.)
2 is close enough. Extrapolating the results of safe tests to unsafe settings requires a level of theoretical competence we don’t currently have. Steve Brynes just made a great post that is somewhat related, I endorse everything in that post.
Thanks. I’ll reread you post again in light of that. I guess I was always assuming that doing that was going to become necessary, as it is in every field, and as is particularly challenging to do safely in safety engineering fields. Also that we were currently only, say O(10%) of the way through the process. So I’m unsurprised we can’t see much of the route yet — but we can see more of it than we could, say, 5 years ago, and usually we do figure these things out eventually. What concerns me isn’t that this is impossible, but that I don’t think we’re on track to be done in another, say, 5 years, that rushing is is a great way to cause 3.a., and it’s unclear how long we have left.
I’ll also take a look at Steve Bryrnes post you link.
I’m not claiming it’s conceptually complex. The conceptual simplicity is a large part of the motivation for pursuing this. Some of the difficulty comes up when you try to design the implementation details and need it to also be reflectively stable and generalise appropriately. The other difficulty comes from different implementations being difficult to distinguish from a training perspective, but having different reflective stability and generalisation properties.
That was not ever the intended meaning of the word. Using it like that is a great way to confuse everyone.
Both of these terms have different meanings from what I intend. AI-assisted alignment is closely related, but it is a process made possible by stable correctability, not the same thing as stable correctablility. Having a term for stable correctability itself is useful.
I’m aware that there are certain kinds of almost-aligned that will correct themselves. But I don’t see this as very relevant, that kinds of brokenness was never the important kind, the important kind of design-error are the ones that affect a reflectively consistent degree of freedom.
You misunderstand my intended thesis, it’s not related to quantity of work, it’s closer to how much “ability to course-correct” that humans have.
Let me make sure I’ve understood you. The AI does something, mistakenly thinking that it’ll make me happy. I am not happy, and tell it “Hey! Never do that again!”. The AI Bayesian updates its model of what humans like in light of this new data. How is that not correctly described as “corrigibility” (the ability to be corrected)? I corrected the agent, it incorporated this data appropriately into its model of how the world and humans work (including properly allowing for possibilities like that I was confused, drunk, or actually upset about something else, rather than simply taking my word for it because I’m human). That seems like, not just corrigibility, but exactly the ideal level of corrigibility. Are you using corrigibility in some sense other than “the humans have the capability to correct the AI’s mistaken beliefs about what its utility function/goals should be”? If so, perhaps you should define it?
Or are you concerned that the AI will run out of belief that it could be mistaken, or at least could have more to learn, too early: its posteriors will all irreversibly reach 99.9999999…% with no possibility that it needs to consider any previous unconsidered hypotheses about anything about human values or how best to optimized them before its beliefs about human values are actually functionally complete — i.e. that it’ll be “a bad Bayesian”? (Personally I tend to assume that bad Bayesians are not actually going to be able to take over the world and go Foom, but YMMV.)
I must admit, I did find your post hard to penetrate. The pessimism came across, the “but we can only test in toy situations” analogy about ships — but it wasn’t clear to me what you meant by “this will run out of fuel”, and I didn’t get why you appear to be assuming the AI can’t help, in a post about “a basin of attraction”. IMO, for us to be in a basin of attraction, we need to already have nearly-aligned, significantly-capable AI, that wants to help with alignment, can help with alignment, and actually will (mostly) help with alignment. And the requirement for convergence is that, net between us plus the AI, we continue on average to make forward progress, so the process converges rather than diverging. That doesn’t have to run out of fuel because we reached the edge of what humans could do unaided — but it does require that then the aid we’re getting is sufficiently trustworthy (at least after we’ve examined it carefully) to cause convergence rather than divergence.
That’s corrigible behaviour, but the mechanism is not, because it stops being corrigible after some number of updates. The idea of corrigibility is that it is correctable, and remains so, in spite of designers making mistakes in goal alignment (or other design mistakes). (Of course mistakes in however we enforced the corrigibility property itself might not be stably correctable, but the hope is that this part is easier to get right on the first try than the rest of it).
Often asserted, but simply untrue. Bayesian updates never stop being corrigible. If the sun stops rising in the East, people update. After a few failed sunrises, they update hard. Bayesian posteriors have the Martingale property: their future direction of change is not predictable from their current value (if it were, you should already have updated in that direction). So even if it’s very high, it’s not just possible but probable (a-priori, has a 50% chance) that the next update will be a drop. (For approximate Bayesian reasoners, this remains true, unless you have access to significantly more computational capacity than it does — a smarter agent may see something it missed.) It takes a mountain of evidence to drive a Bayesian posterior very high, but an equally large mountain of opposing evidence will always drive it right back down again. Or, more often, even a small hill of opposing evidence will cause a good approximate Bayesian to spawn a new hypothesis that it hadn’t previously considered, one more compatible with both the mountain and the hill of evidence than “two huge/large opposing coincidences occurred”. E.g. a vast amount of evidence supporting Newtonian Mechanics doesn’t disprove Relativity, if it’s all from situations where they give indistinguishable results. In general, if I give an approximate Bayesian reasoner even ~30-bits-worth of opposing evidence, it should start looking for new hypotheses rather than just assuming that it’s right and a one-in-a-billion coincidence has just occurred.
[If it doesn’t do this, it’s a worse Bayesian than humans are, and thus hopefully not that dangerous — if a conflict occurred, we could outsmart it.]
Not what I meant. See Fully Update Deference.
I’m familiar with the argument. I just don’t agree with it. I think fully updated deference is asking for the impossible: you want a rational agent to continue letting you change your mind (and its) indefinitely, and not ever come to the obvious conclusion that you aren’t telling it the truth or are in fact confused. Personally, I’m willing to accept that a human-or-better approximate Bayesian reasoner rationally deducing what human values are eventually will do at least as good a job as we can by correcting it, and will be aware of this, and thus will eventually stop giving us deference beyond simply treating us as a source of data points, other than out of politeness. So (to paraphrase a famous detective), having eliminated the impossible, whatever is left I am willing to term “corrigibility”. If your definition of “corrigibility” includes fully updated deference, then yes, I agree, it’s impossible to achieve on the basis of Bayesian uncertainty: the Bayesian will eventually realize you’re being unreasonable, if you make enough unreasonable demands, and stop listening to you. However, if you only correct it with good reason, and it’s a good Bayesian, then you won’t run out of corrigibility.
In short, I’m unwilling to accept redefining the everyday term “corrigibility” to include “something logically impossible”, and then saying that they’ve proven that corrigibility is impossible — it’s linguistic slight of hand. I would suggest instead creating a more accurate term, such as saying something like that “unreasonably-unlimited corrigibility” isn’t possible on the basis of Bayesian uncertainty. Which is, well, unsurprising.
Returning to your concern that we may “run out of fuel” — only if we waste it by making unreasonable demands. We have all the corrigibility we could actually need — a good Bayesian isn’t going to decide we’re untrustworthy and stop paying us deference unless we actually do something clearly untrustworthy, like expect the right to keep changing our mind indefinitely.
Also, this does mean “reasonable”: if society changed and the AI just went out of distribution, and is wrong as a result, we get to tell it so, and it should keep spawning new hypotheses and collecting new evidence until it’s fully updated in this distribution region as well — thus my discussion above of Relativity.
This also generalizes, in ways that I think do actually give you something pretty close to fully updated deference. Just as if the sun stopped rising in the East, people would update, if you give a “fully updated” Bayesian reasoner ~30 bits of evidence that you want to shut it down (for reasons that actually look like it’s made a mistake and you’re legitimately scared and not confused, rather than some obvious other human motive), it should say: either a vastly improbably one-in-a-billion event has just occurred, or there’s a hypothesis missing from my hypothesis space. The latter seems more plausible. Maybe what humans value just changed, or there’s something I’ve been missing so far? Let’s spawn some new hypotheses, consistent with both all the old data and this new “ought to be incredible improbable” observation. It sure looks like they’re really scared. I talked to them about this, and they don’t appear to simply be confused… (And if it doesn’t say that, give it another ~30 bits of evidence.)
I see that you’re doing large edits and additions to your previous responses, after I had already responded.
This, and the way you’re playing with definitions, makes me think you might be arguing in bad faith. I’m going to stop responding. If you had good intentions, I’m sorry.
I did have good intentions, I just like to make my exposition as clear and well-thought-through as possible. But that’s fine, I think we have rather different views, and you’re under no obligation to engage with mine. Your alternative would be to wait until my reply stabilizes before replying to it, which is generally O(an hour). Remaining typo density is another cue. Sadly there is no draft option on the replies, and I can’t be bothered to do the editing somewhere else. Most interloctors don’t reply as quickly as you have been, so this hasn’t previously caused problems that I’m aware of.
On “playing with definitions” — actually, I’m saying that, IMO, some thinkers associated with MIRI have done so (see the second paragraph of my previous post).