I think I basically agree with all this, pace the parenthetical that I of course approach more dubiously.
But I like the explicit spelling out that “processes capable of achieving ends are coherent over time” is very different from “minds (sub-parts of processes) that can be part of highly-capable actions will become more coherent over time.”
A mind’s long-term behavior is shaped by whichever of its shards have long-term goals, because shards that don’t coherently pursue any goal end up, well, failing to have optimized for any goal over the long term.
If the internal shards with long-term goals are the only thing shaping the long-term evolution of the mind, this looks like it’s so?
But that’s a contingent fact—many things could shape the evolution of minds, and (imo) the evolution of minds is generally dominated by data and the environment rather than whatever state the mind is currently in. (The environment can strength some behaviors and not others; shards with long-term goals might be less friendly to other shards, which could lead to alliances against them; the environment might not even reward long-horizon behaviors, vastly strengthening shorter-term shards; you might be in a social setting where people distrust unmitigated long-term goals without absolute deontological short-term elements; etc etc etc)
(...and actually, I’m not even really sure it’s best to think of “shards” as having goals, either long-term or short-term. That feels like a confusion to me maybe? a goal is perhaps the result of a search for action, and a “shard” is kinda a magical placeholder for something generally less complex than the search for an action.)
...and actually, I’m not even really sure it’s best to think of “shards” as having goals, either long-term or short-term
Agreed; I was speaking loosely. (One line of reasoning there goes: shards are contextually activated heuristics; heuristics can be viewed as having been optimized for achieving some goal; inspecting shards (via e. g. self-reflection) can lead to your “reverse-engineering” those implicitly encoded goals; therefore, shards can be considered “proto-goals/values” of a sort, and complex patterns of shard activations can draw the rough shape of goal-pursuit.)
I think I basically agree with all this, pace the parenthetical that I of course approach more dubiously.
But I like the explicit spelling out that “processes capable of achieving ends are coherent over time” is very different from “minds (sub-parts of processes) that can be part of highly-capable actions will become more coherent over time.”
If the internal shards with long-term goals are the only thing shaping the long-term evolution of the mind, this looks like it’s so?
But that’s a contingent fact—many things could shape the evolution of minds, and (imo) the evolution of minds is generally dominated by data and the environment rather than whatever state the mind is currently in. (The environment can strength some behaviors and not others; shards with long-term goals might be less friendly to other shards, which could lead to alliances against them; the environment might not even reward long-horizon behaviors, vastly strengthening shorter-term shards; you might be in a social setting where people distrust unmitigated long-term goals without absolute deontological short-term elements; etc etc etc)
(...and actually, I’m not even really sure it’s best to think of “shards” as having goals, either long-term or short-term. That feels like a confusion to me maybe? a goal is perhaps the result of a search for action, and a “shard” is kinda a magical placeholder for something generally less complex than the search for an action.)
Agreed; I was speaking loosely. (One line of reasoning there goes: shards are contextually activated heuristics; heuristics can be viewed as having been optimized for achieving some goal; inspecting shards (via e. g. self-reflection) can lead to your “reverse-engineering” those implicitly encoded goals; therefore, shards can be considered “proto-goals/values” of a sort, and complex patterns of shard activations can draw the rough shape of goal-pursuit.)