“Let’s say my wool sweater is wet but I want to wear it outside when I leave in 10 minutes. So I have a goal (look nice tonight) and a subgoal (dry my sweater within 10 minutes). Some “module” in my brain is focused on the subgoal, and one thing that could happen is: it finds that there’s really only one way to accomplish the subgoal, and it’s to put the sweater in the dryer on the highest possible heat. It then notices that this will be shrink the sweater to look terrible, which is a problem according to the “look nice tonight” goal, so the “module” “solves” that problem by simply deleting the “look nice tonight” goal! …This scenario doesn’t actually happen, so that calls for an explanation of why not, and that explanation (whatever it is) will be a solution to corrigibility.”
Assuming that’s what you meant: I think the reason the scenario doesn’t happen (in humans) is because “modules” is not as literal a thing as you make it out to be. A better analogy is, like, looking at a big painting. You can’t take in the whole thing at once (among other things, your vision is only sharp at the fovea), so you focus on one part, and then another part, etc. But whatever you’re focusing on right now, you’re interpreting it in the context of the rest of the painting. That larger context never goes away; it remains active in your mind. And there’s some constraint-satisfaction thing that will ensure that the part of the painting you’re focusing on is being interpreted in a way that’s consistent with that larger context. Or if an inconsistency arises, your attention will be drawn to it—it will jump out with a feeling of confusion.
By the same token, when you’re making a plan, you will be (at some points in serial time) be focusing on the drying-the-sweater problem, but you’ll be thinking about that problem in the context of the larger plan that involves looking nice tonight. Looking nice tonight remains at the back of your mind the whole time, throwing out demands into the constraint-satisfaction soup. And the idea of looking nice tonight is the only thing you ACTUALLY care about, in terms of what the valence (a.k.a. value function, a.k.a. critic) will upvote versus downvote, so that part will win when deciding tradeoffs.
Anyway, does this suggest a solution to AI corrigibility? Well, kinda. But it’s just the obvious one: the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind, and notices when it has an idea that would bother or offend the supervisor, and then not do it. I do talk about that briefly in §4.2, which cites my previous post “Act-based approval-directed agents”, for IDA skeptics.
The sweater example is close but doesn’t quite hit the nail on the head. It’s not that the (dry sweater) planner would delete the (look nice) goal; that wouldn’t help dry the sweater! Rather, the (dry sweater) planner would try to commandeer more mental resources, i.e. more planner-modules, steer more attention to drying the sweater. That additional attention potentially helps dry the sweater. But as an accidental side-effect, attention would be steered away from other goals, like e.g. (look nice). In short, the (dry sweater) planner-module can better dry the sweater by redirecting the (look nice) module to focus on drying the sweater instead.
… and that totally does happen in humans! Humans often get caught up in a specific subgoal, lose track of the broader goal which generated that subgoal in the first place, and end up optimizing for the subgoal in a way which doesn’t help the original goal. It’s the phenomenon of lost purposes, at an individual level.
(Likewise with the painting example: when looking at little patches of a large complex painting, people will totally lose track of context and overlook inconsistencies.)
It really doesn’t seem like humans “keep their eye on the ball” all the time, even in the large majority of day-to-day cases where cognition basically works.
We can divide things into an inference algorithm (what to do now) and a learning algorithm (how to change stored parameters such that I do better in the future). These correspond respectively to search / planning / foresight, and to RL / updating-from-mistakes-and-surprises. In humans, generally both the inference algorithm and learning algorithm are working together throughout life. (Although in the ASI context, as intelligence and knowledge goes up, we expect foresight to get better, and thus fewer mistakes and surprises, and thus the balance tilts towards the inference algorithm over the learning algorithm.)
So yeah, sure, people will sometimes pursue a subgoal while losing track of the actual goal, because the inference algorithm is imperfect. But then the person would later on notice that they failed to achieve their actual goal, and see that as a bad thing, and then the learning algorithm will kick in and help them avoid a similar mistake next time.
This system is not perfect, but we generally get by, especially in familiar situations.
I still don’t think there’s any lesson here for AI corrigibility, except what I said before: “the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind, and notices when it has an idea that would bother or offend the supervisor, and then not do it”. I guess that sentence was only discussing the inference-algorithm part, and I omitted the corresponding learning-algorithm part, which would be: “…and also, the AI notices if it violates the supervisor’s desires / preferences / aversions despite its intentions, and when that happens, the AI feels bad and thinks about how to do better next time”.
This (the inference-algorithm part and learning-algorithm part together) comprises an approach to corrigibility that I think many people treat as the obvious default plan.
“You can’t take in the whole thing at once (among other things, your vision is only sharp at the fovea), so you focus on one part, and then another part, etc.”
That is, in a sense, not entirely true. Paintings can be like eyesight on purpose. Impressionism (or some “schools” of Impressionism) is “about” that, really. The idea of having a whole scenario unfold in front of you, all things at once, and because of the retinal limitations of focus, think about what is a “pusher” in a painting, and paint it according what you want your leading effects to be, what you have identified as such. Precisely to not noodle up a collage of single items and render them generically (detailed). You are interpreting the rest of the painting anyway, but you have leading effects.
Anyway, does this suggest a solution to AI corrigibility? Well, kinda. But it’s just the obvious one: the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind, and notices when it has an idea that would bother or offend the supervisor, and then not do it.
I think it totally does suggest a solution, if you could reformulate the quote in a much more abstract and general way, along with explaining how/why the AI doesn’t fool the supervisor. I was skimming the intro to brain-like AGI safety and IIRC the problem “how/why doesn’t the thought generator fool the thought accessor?” wasn’t solved beyond “create as many thought accessors as possible and hope for the best”.
In general, I’m very interested in the intersection between brain-like ideas and agent foundations ideas.
I was skimming the intro to brain-like AGI safety [link added] and IIRC the problem “how/why doesn’t the thought generator fool the thought accessor?” wasn’t solved beyond “create as many thought accessors as possible and hope for the best”.
This is off-topic here but happy to chat about it anyway. The thought generator generates thoughts, and the thought assessor judges those thoughts as good or bad. Neither of these modules is an intelligent agent, so I don’t know what it means for the thought generator to “fool” the thought assessor. It’s not like the thought assessor has goals and aspirations which can be undermined; rather, the thought assessor is just a machine that stamps “good” or “bad” on different thoughts. Is it possible for thoughts to get stamped “good” or “bad” for weird reasons? Absolutely! We might call that “reward hacking” (although what constitutes reward hacking is in the eye of the beholder), and it is ubiquitous in everyday human life. (See my discussion under the heading “The arc of progress is long, but it bends towards reward hacking”.) If you ask me “what’s the solution to reward hacking in brain-like AGI?”, then my answer is “I don’t know”. I continue to brainstorm, and sometimes share my ill-formed half-baked thoughts in posts like this one! :) See e.g. My AGI safety research—2025 review, ’26 plans.
Sorry, why is this offtopic—what are the different problems I could’ve confused? The thing I quoted feels like it assumes exactly the reward hacking problem being solved:
the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind, and notices when it has an idea that would bother or offend the supervisor, and then not do it
It assumes we have a thing which can accurately judge ideas as “good” or “bad”. I don’t believe it’s a solved problem. I went on to re-read the “act-based approval-directed agents” post, but it offered no solution. This reply to John seems to draw a distinction between good foresight and good hindsight, but I’d say both are unsolved problems and might have a similar solution. The latter is the credit assignment problem and I remember your post about it, with no general solution.
I’ll share my general view, maybe it’ll help to sort out the misunderstanding:
I think the brain is a horrible mess. “Why does the whole system stay relatively stable, without drifting uncontrollably into arbitrary directions?” is the core alignment-related mystery about it. “How is the brain able to correctly introspect its thoughts?”, “how is the brain able to notice failures/successes of its high-level goals?”, etc. are all facets of the core mystery. I believe they are all unsolved and not obvious. IMO any solution to any of the facets will give a (non-trivial) insight about corrigibility.
When you keep saying “it’s not a solved problem” and “no solution”, are you saying that there are important mysteries about how the human brain works, or that there are important mysteries about how to solve the technical alignment problem for brain-like AGI?
If the latter, yes I strongly agree, I say that all the time. If you think I have ever claimed to know a good plan for technical alignment (or corrigibility etc.) then either I made a typo or you misunderstood me. For example, when I wrote “the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind …”, the context was a discussion of high-level desiderata / research directions, not nuts-and-bolts technical plans.
If the former (if you’re saying that there are important mysteries about how the human brain works), then I’m confused that you’re making this comment in a conversation about AI alignment, and linking to other posts on AI alignment. There’s a question of whether or not my neuro framework can explain every aspect of human psychology / behavior / experience, and that question is unrelated to AI alignment. (I obviously think the answer is yes, and if you disagree then we can talk about the aspects of human experience that you think it can’t explain, but please leave AI out of that conversation.)
I’d say “we don’t understand how the brain works good enough to replicate it in AGI” (and that means we lack a really important layer of understanding). Anyway, I think we agree here.
The point I wanted to make:
I think John’s argument for why corrigibility is natural works even if you don’t assume crisp separate modules.
Whatever explanation you propose (of the brain staying coherent), be it “your modules are corrigible to each other” or “the goals are kept at the back of your mind”, if you try to make that explanation more technical you’ll probably end up with a general solution to corrigibility.
Sorry for skipping so much inferential steps and causing confusion.
I interpret you as saying:
“Let’s say my wool sweater is wet but I want to wear it outside when I leave in 10 minutes. So I have a goal (look nice tonight) and a subgoal (dry my sweater within 10 minutes). Some “module” in my brain is focused on the subgoal, and one thing that could happen is: it finds that there’s really only one way to accomplish the subgoal, and it’s to put the sweater in the dryer on the highest possible heat. It then notices that this will be shrink the sweater to look terrible, which is a problem according to the “look nice tonight” goal, so the “module” “solves” that problem by simply deleting the “look nice tonight” goal! …This scenario doesn’t actually happen, so that calls for an explanation of why not, and that explanation (whatever it is) will be a solution to corrigibility.”
Assuming that’s what you meant: I think the reason the scenario doesn’t happen (in humans) is because “modules” is not as literal a thing as you make it out to be. A better analogy is, like, looking at a big painting. You can’t take in the whole thing at once (among other things, your vision is only sharp at the fovea), so you focus on one part, and then another part, etc. But whatever you’re focusing on right now, you’re interpreting it in the context of the rest of the painting. That larger context never goes away; it remains active in your mind. And there’s some constraint-satisfaction thing that will ensure that the part of the painting you’re focusing on is being interpreted in a way that’s consistent with that larger context. Or if an inconsistency arises, your attention will be drawn to it—it will jump out with a feeling of confusion.
By the same token, when you’re making a plan, you will be (at some points in serial time) be focusing on the drying-the-sweater problem, but you’ll be thinking about that problem in the context of the larger plan that involves looking nice tonight. Looking nice tonight remains at the back of your mind the whole time, throwing out demands into the constraint-satisfaction soup. And the idea of looking nice tonight is the only thing you ACTUALLY care about, in terms of what the valence (a.k.a. value function, a.k.a. critic) will upvote versus downvote, so that part will win when deciding tradeoffs.
Anyway, does this suggest a solution to AI corrigibility? Well, kinda. But it’s just the obvious one: the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind, and notices when it has an idea that would bother or offend the supervisor, and then not do it. I do talk about that briefly in §4.2, which cites my previous post “Act-based approval-directed agents”, for IDA skeptics.
The sweater example is close but doesn’t quite hit the nail on the head. It’s not that the (dry sweater) planner would delete the (look nice) goal; that wouldn’t help dry the sweater! Rather, the (dry sweater) planner would try to commandeer more mental resources, i.e. more planner-modules, steer more attention to drying the sweater. That additional attention potentially helps dry the sweater. But as an accidental side-effect, attention would be steered away from other goals, like e.g. (look nice). In short, the (dry sweater) planner-module can better dry the sweater by redirecting the (look nice) module to focus on drying the sweater instead.
… and that totally does happen in humans! Humans often get caught up in a specific subgoal, lose track of the broader goal which generated that subgoal in the first place, and end up optimizing for the subgoal in a way which doesn’t help the original goal. It’s the phenomenon of lost purposes, at an individual level.
(Likewise with the painting example: when looking at little patches of a large complex painting, people will totally lose track of context and overlook inconsistencies.)
It really doesn’t seem like humans “keep their eye on the ball” all the time, even in the large majority of day-to-day cases where cognition basically works.
Thanks!
We can divide things into an inference algorithm (what to do now) and a learning algorithm (how to change stored parameters such that I do better in the future). These correspond respectively to search / planning / foresight, and to RL / updating-from-mistakes-and-surprises. In humans, generally both the inference algorithm and learning algorithm are working together throughout life. (Although in the ASI context, as intelligence and knowledge goes up, we expect foresight to get better, and thus fewer mistakes and surprises, and thus the balance tilts towards the inference algorithm over the learning algorithm.)
So yeah, sure, people will sometimes pursue a subgoal while losing track of the actual goal, because the inference algorithm is imperfect. But then the person would later on notice that they failed to achieve their actual goal, and see that as a bad thing, and then the learning algorithm will kick in and help them avoid a similar mistake next time.
This system is not perfect, but we generally get by, especially in familiar situations.
I still don’t think there’s any lesson here for AI corrigibility, except what I said before: “the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind, and notices when it has an idea that would bother or offend the supervisor, and then not do it”. I guess that sentence was only discussing the inference-algorithm part, and I omitted the corresponding learning-algorithm part, which would be: “…and also, the AI notices if it violates the supervisor’s desires / preferences / aversions despite its intentions, and when that happens, the AI feels bad and thinks about how to do better next time”.
This (the inference-algorithm part and learning-algorithm part together) comprises an approach to corrigibility that I think many people treat as the obvious default plan.
A little painting-sidenote:
“You can’t take in the whole thing at once (among other things, your vision is only sharp at the fovea), so you focus on one part, and then another part, etc.”
That is, in a sense, not entirely true. Paintings can be like eyesight on purpose. Impressionism (or some “schools” of Impressionism) is “about” that, really. The idea of having a whole scenario unfold in front of you, all things at once, and because of the retinal limitations of focus, think about what is a “pusher” in a painting, and paint it according what you want your leading effects to be, what you have identified as such. Precisely to not noodle up a collage of single items and render them generically (detailed). You are interpreting the rest of the painting anyway, but you have leading effects.
I think it totally does suggest a solution, if you could reformulate the quote in a much more abstract and general way, along with explaining how/why the AI doesn’t fool the supervisor. I was skimming the intro to brain-like AGI safety and IIRC the problem “how/why doesn’t the thought generator fool the thought accessor?” wasn’t solved beyond “create as many thought accessors as possible and hope for the best”.
In general, I’m very interested in the intersection between brain-like ideas and agent foundations ideas.
This is off-topic here but happy to chat about it anyway. The thought generator generates thoughts, and the thought assessor judges those thoughts as good or bad. Neither of these modules is an intelligent agent, so I don’t know what it means for the thought generator to “fool” the thought assessor. It’s not like the thought assessor has goals and aspirations which can be undermined; rather, the thought assessor is just a machine that stamps “good” or “bad” on different thoughts. Is it possible for thoughts to get stamped “good” or “bad” for weird reasons? Absolutely! We might call that “reward hacking” (although what constitutes reward hacking is in the eye of the beholder), and it is ubiquitous in everyday human life. (See my discussion under the heading “The arc of progress is long, but it bends towards reward hacking”.) If you ask me “what’s the solution to reward hacking in brain-like AGI?”, then my answer is “I don’t know”. I continue to brainstorm, and sometimes share my ill-formed half-baked thoughts in posts like this one! :) See e.g. My AGI safety research—2025 review, ’26 plans.
Sorry, why is this offtopic—what are the different problems I could’ve confused? The thing I quoted feels like it assumes exactly the reward hacking problem being solved:
It assumes we have a thing which can accurately judge ideas as “good” or “bad”. I don’t believe it’s a solved problem. I went on to re-read the “act-based approval-directed agents” post, but it offered no solution. This reply to John seems to draw a distinction between good foresight and good hindsight, but I’d say both are unsolved problems and might have a similar solution. The latter is the credit assignment problem and I remember your post about it, with no general solution.
I’ll share my general view, maybe it’ll help to sort out the misunderstanding:
I think the brain is a horrible mess. “Why does the whole system stay relatively stable, without drifting uncontrollably into arbitrary directions?” is the core alignment-related mystery about it. “How is the brain able to correctly introspect its thoughts?”, “how is the brain able to notice failures/successes of its high-level goals?”, etc. are all facets of the core mystery. I believe they are all unsolved and not obvious. IMO any solution to any of the facets will give a (non-trivial) insight about corrigibility.
When you keep saying “it’s not a solved problem” and “no solution”, are you saying that there are important mysteries about how the human brain works, or that there are important mysteries about how to solve the technical alignment problem for brain-like AGI?
If the latter, yes I strongly agree, I say that all the time. If you think I have ever claimed to know a good plan for technical alignment (or corrigibility etc.) then either I made a typo or you misunderstood me. For example, when I wrote “the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind …”, the context was a discussion of high-level desiderata / research directions, not nuts-and-bolts technical plans.
If the former (if you’re saying that there are important mysteries about how the human brain works), then I’m confused that you’re making this comment in a conversation about AI alignment, and linking to other posts on AI alignment. There’s a question of whether or not my neuro framework can explain every aspect of human psychology / behavior / experience, and that question is unrelated to AI alignment. (I obviously think the answer is yes, and if you disagree then we can talk about the aspects of human experience that you think it can’t explain, but please leave AI out of that conversation.)
I’d say “we don’t understand how the brain works good enough to replicate it in AGI” (and that means we lack a really important layer of understanding). Anyway, I think we agree here.
The point I wanted to make:
I think John’s argument for why corrigibility is natural works even if you don’t assume crisp separate modules.
Whatever explanation you propose (of the brain staying coherent), be it “your modules are corrigible to each other” or “the goals are kept at the back of your mind”, if you try to make that explanation more technical you’ll probably end up with a general solution to corrigibility.
Sorry for skipping so much inferential steps and causing confusion.