At a lower level, motivations could be structured in terms of myopic circuits (certain LLM-isms incentivized when engaging with reward models, task reward hacking, apparent-success-seeking) and non-myopic circuits (HHH behavior?), where the degree to which myopic preference circuits are upweighted—or overrule non-myopia—depend on the scope of the task environment. Identifying motivation-relevant circuits might also scale with parameter decomposition
But that framing doesn’t really consider agency- maybe in some sense Claude is closer to an embedded agent with preference circuits acting as subsystems
(assuming you’re working on an extension of this post?)
At a lower level, motivations could be structured in terms of myopic circuits (certain LLM-isms incentivized when engaging with reward models, task reward hacking, apparent-success-seeking) and non-myopic circuits (HHH behavior?), where the degree to which myopic preference circuits are upweighted—or overrule non-myopia—depend on the scope of the task environment. Identifying motivation-relevant circuits might also scale with parameter decomposition
But that framing doesn’t really consider agency- maybe in some sense Claude is closer to an embedded agent with preference circuits acting as subsystems
(assuming you’re working on an extension of this post?)