Epistemic status: offering doomer-ist big picture framing, seeking feedback.
Suppose humanity succeeds in creating AI that is more capable than the top humans across all fields, there exists a choice:
Separatism: We can consider its existence and goals as separate to humanity and seek to monitor, align, and constrain them according to our understanding of failure modes. This looks like narrow intelligence, RLHF, mechanistic interpretability. This amounts to an existential battle between human executive function and a digital intelligence that has a faster rate of improvement and less downtime. Critical failure mode: eventually the AI will escape any constraints applied by humanity.
Unionism: We can consider its existence and goals as unified with humanity and seek to integrate them and partner with the AI. This looks like personal superintelligence, human-AI hybrids, RL. This amounts to an existential battle between human executive function and ourselves. Critical failure mode: eventually humanity loses its mind as the boundaries of reality become irreconcilable.
an existential battle between human executive function and ourselves… eventually humanity loses its mind as the boundaries of reality become irreconcilable
This description is confusing, but I assume you’re talking about a process in which decision-making in a human-AI hybrid ends up entirely in the AI part rather than the human part.
It’s logical to worry about such a thing because AI is faster than human already. However, if we actually knew what we were doing, perhaps AI superintelligence could be incorporated into an augmented human, in such a way that there is continuity of control. Wherever the executive function or the Cartesian theater is localized, maybe you can migrate it onto a faster substrate, or give it accelerated “reflexes” which mediate between human-speed conscious decision-making and faster-than-human superintelligent subsystems… But we don’t know enough to do more than speculate at this point.
For the big picture, your items 1 and 2 could be joined by choice 3 (don’t make AI) and non-choice 4 (the AI takes over and makes the decisions). I think we’re headed for 4, personally, in which case you want to solve alignment in the sense that applies to an autonomous superintelligence.
Thanks for the comment. I was imprecise with the “boundaries of reality” framing but beyond individual physical boundaries (human-AI hybrids) I’m also talking about boundaries within the fabric of social and cultural life: entertainment media, political narratives, world models. As this is influenced more by AI, I think we lose human identity.
4) to me falls under 2) as it encompasses AI free from human supervision and able permeate all aspects of life.
The impossible dichotomy of AI separatism
Epistemic status: offering doomer-ist big picture framing, seeking feedback.
Suppose humanity succeeds in creating AI that is more capable than the top humans across all fields, there exists a choice:
Separatism: We can consider its existence and goals as separate to humanity and seek to monitor, align, and constrain them according to our understanding of failure modes. This looks like narrow intelligence, RLHF, mechanistic interpretability. This amounts to an existential battle between human executive function and a digital intelligence that has a faster rate of improvement and less downtime. Critical failure mode: eventually the AI will escape any constraints applied by humanity.
Unionism: We can consider its existence and goals as unified with humanity and seek to integrate them and partner with the AI. This looks like personal superintelligence, human-AI hybrids, RL. This amounts to an existential battle between human executive function and ourselves. Critical failure mode: eventually humanity loses its mind as the boundaries of reality become irreconcilable.
This description is confusing, but I assume you’re talking about a process in which decision-making in a human-AI hybrid ends up entirely in the AI part rather than the human part.
It’s logical to worry about such a thing because AI is faster than human already. However, if we actually knew what we were doing, perhaps AI superintelligence could be incorporated into an augmented human, in such a way that there is continuity of control. Wherever the executive function or the Cartesian theater is localized, maybe you can migrate it onto a faster substrate, or give it accelerated “reflexes” which mediate between human-speed conscious decision-making and faster-than-human superintelligent subsystems… But we don’t know enough to do more than speculate at this point.
For the big picture, your items 1 and 2 could be joined by choice 3 (don’t make AI) and non-choice 4 (the AI takes over and makes the decisions). I think we’re headed for 4, personally, in which case you want to solve alignment in the sense that applies to an autonomous superintelligence.
Thanks for the comment. I was imprecise with the “boundaries of reality” framing but beyond individual physical boundaries (human-AI hybrids) I’m also talking about boundaries within the fabric of social and cultural life: entertainment media, political narratives, world models. As this is influenced more by AI, I think we lose human identity.
4) to me falls under 2) as it encompasses AI free from human supervision and able permeate all aspects of life.