Unionism sound like two things pretending to be one thing. Does “blend” mean we get more of their values, or they get more of ours? I’m reminded of company mergers, where the smaller company basically ends up getting eaten, despite initial promises to the contrary.
To handle this I think we would consider two traits:[2]
Agency permeability:[3] every tier of cognition can both influence and be influenced by the global action policy[4]with low friction or external arbitration (e.g driving your car to work — switching between intentional decisions and muscle memory)
Homeostatic unity:all subsystems participate in the same self-maintenance goal (e.g biological survival and personal welfare)
For the unified entity (i.e human & superintelligence (SI)) to be well aligned we would ensure that they are unified such that these traits are protected.
I propose an example implementation for unification:
I used a few back-and-forths with an LLM, plus gathered feedback, to find a criteria and definition that could work here.
Initially this was “unity of agency: one locus of practical authority”, but this seems to 1) repeat ideas from normative closure, and 2) fail to capture how agency almost free-associates between layers of cognition in a human. We have times when we are “active and intentional” in our thoughts (i.e locus of control towards second-order) vs. operating on muscle memory (e.g, when driving a car) where the locus of control is closer to first-order.
“I first need to determine a point of observability for the metaphysical conditions — the current moment. I can then define the global action policy for a being as its “master mapping from its current informational state (internal and external) to enacted behaviour, after all internal competitions and vetoes have been resolved”. The action may be something like eating a marshmellow.”
Unionism sound like two things pretending to be one thing. Does “blend” mean we get more of their values, or they get more of ours? I’m reminded of company mergers, where the smaller company basically ends up getting eaten, despite initial promises to the contrary.
It’s a good concern, and I address this in my third-order cognition manifesto.[1]
To handle this I think we would consider two traits:[2]
For the unified entity (i.e human & superintelligence (SI)) to be well aligned we would ensure that they are unified such that these traits are protected.
I propose an example implementation for unification:
I initially referred to this as a thesis, but I think it’s a bit more pointed (and meandering at times).
The manifesto contains 8 traits, to be focused on for different scenarios.
I used a few back-and-forths with an LLM, plus gathered feedback, to find a criteria and definition that could work here.
Initially this was “unity of agency: one locus of practical authority”, but this seems to 1) repeat ideas from normative closure, and 2) fail to capture how agency almost free-associates between layers of cognition in a human. We have times when we are “active and intentional” in our thoughts (i.e locus of control towards second-order) vs. operating on muscle memory (e.g, when driving a car) where the locus of control is closer to first-order.
“I first need to determine a point of observability for the metaphysical conditions — the current moment. I can then define the global action policy for a being as its “master mapping from its current informational state (internal and external) to enacted behaviour, after all internal competitions and vetoes have been resolved”. The action may be something like eating a marshmellow.”
“Normative closure: a single normative standpoint (e.g stating what you want to do)”
“Persistence conditions: criteria for maintaining identity as the same being over change (e.g. losing a limb)”
“Boundary conditions: clearly defined membership rules on what binds the being (e.g having skin and bones)”