My sense is that it’s good to have a distinct word for “context → external action” mappings, and a different word for “context → actions+internal state changes”. We want the AI to ignore the human’s thoughts, and not treat them as actions, but then also still track that they’re part of what minds do...
Why do we want to ignore them? (Maybe you think that there is no gap between “do not ignore” and “do not attempt to modify”, but I am not sure this is so true.)
Ignore might have been the wrong word. Mainly I think it’s a really bad situation if the AI is taking the human’s unreflective thoughts as directives, rather than giving the human space to think about what’s best before directing the AI. If the AI responds, for example, to words in my internal verbal loop, then I worry that I cannot actually control it well.
Yes? And I’m noting that both (maintaining the distinction) exist in the output channel?
But you define policy as a mapping between contexts and actions, not as a mapping between contexts and actions+thoughts.
Ah, good point. That’s an inconsistency!
My sense is that it’s good to have a distinct word for “context → external action” mappings, and a different word for “context → actions+internal state changes”. We want the AI to ignore the human’s thoughts, and not treat them as actions, but then also still track that they’re part of what minds do...
Why do we want to ignore them? (Maybe you think that there is no gap between “do not ignore” and “do not attempt to modify”, but I am not sure this is so true.)
Ignore might have been the wrong word. Mainly I think it’s a really bad situation if the AI is taking the human’s unreflective thoughts as directives, rather than giving the human space to think about what’s best before directing the AI. If the AI responds, for example, to words in my internal verbal loop, then I worry that I cannot actually control it well.
Well, yes, but the fact that we want to avoid a certain kind of reactions to thoughts does not entail we should lose all reactions to thoughts.