Rewards and Utilities are different concepts. To reject that reward is necessary to get/build agency is not the same thing as rejecting EU maximization as a basin of idealized agency.
particlemania
Karma: 122
Announcing “Key Phenomena in AI Risk” (facilitated reading group)
As an addendum, it seems to me that you may not necessarily need a ‘long-term planner’ (or ‘time-unbounded agent’) in the environment. A similar outcome may also be attainable if the environment contains a tiling of time-bound agents who can all trade across each other in ways such that the overall trade network implements long term power seeking.
Reflections on the PIBBSS Fellowship 2022
Concept Dictionary.
Concepts that I intend to use or invoke in my writings later, or are parts of my reasoning about AI risk or related complex systems phenomena.
My understanding of Steel Late Wittgenstein’s response would be that you could agree with that words and concepts are distinct, and mapping is not always 1-1, but that what concepts get used is also significantly influenced by which features of the world are useful in some contexts of language (/word) use.