This is a beautiful and clarifying post, which I found just as thrilling to read as I did janus’s original Simulators post—a high bar. Thank you!
Many comments come to mind. I’ll start with one around the third core claim in the Introduction: “Unless we manage to coordinate around it, the default outcome is that humanity will eventually be disempowered by a powerful autonomous agent (or agents).” The accompanying graph shows us a point an unknown distance into the future where “Humanity loses control”.
The urgency is correct, but this isn’t the right threat. All three words are wrong: control is too blunt an instrument, you can’t lose what you never had, and humanity has no referent capable of carrying the load we’d like to put on it here.
Humanity doesn’t have control of even today’s AI, but it’s not just AI: climate risk, pandemic risk, geopolitical risk, nuclear risk—they’re all trending to x-risk, and we don’t have control of any of them. They’re all reflections of the same underlying reality: humanity is an infinitely strong infant, with exponentially growing power to imperil itself, but not yet the ability to think or act coherently in response. This is the true threat—we’re in existential danger because our power at scale is growing so much faster than our agency at scale.
This has always been our situation. When we look into the future of AI and see catastrophe, what we’re looking at is not loss of control, but the point at which the rising tide of our power makes our lack of control fatal.
What’s so exciting to me about the cyborgism proposal is that it seems to bear directly on this issue: not just the AI part, all of it. The essence of the current and future LLMs is a collective intelligence they’re learning from our whole species. The nature of the cyborgism proposal is to explore amplified ways of bridging this collective intelligence to individual and collective human agency.
There’s no clear path, but this is the question we need to be asking about simulators and cyborgism. Can they help us scale consciousness and intention the way we’ve already learned to scale power?
The failure modes outlined in the post are daunting, and no doubt there are others. No amount of caution would be too much in pursuing any program involving this type of alien fire. But it’s a mistake to adopt a posture of staying away from the brink—we’re already there.
Which is right in line of Douglas Engelbart’s program for “augmenting human intellect” by “increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems” as outlined in his 1962 paper Augmenting Human Intellect. AI is hopefully the missing piece of the software puzzle, that lets it empower people instead of rigidify life.
Yes, I agree that “humanity loses control” has problems, and I would go further. Buddhists claim that the self is an illusion. I don’t know about that, but “humanity” is definitely an illusion if you’re thinking of it as a single agent, similar to a multicellular creature with a central nervous system. So comparing it to an infant doesn’t seem apt. Whatever it is, it’s definitely plural. An ecosystem, maybe?
This is a beautiful and clarifying post, which I found just as thrilling to read as I did janus’s original Simulators post—a high bar. Thank you!
Many comments come to mind. I’ll start with one around the third core claim in the Introduction: “Unless we manage to coordinate around it, the default outcome is that humanity will eventually be disempowered by a powerful autonomous agent (or agents).” The accompanying graph shows us a point an unknown distance into the future where “Humanity loses control”.
The urgency is correct, but this isn’t the right threat. All three words are wrong: control is too blunt an instrument, you can’t lose what you never had, and humanity has no referent capable of carrying the load we’d like to put on it here.
Humanity doesn’t have control of even today’s AI, but it’s not just AI: climate risk, pandemic risk, geopolitical risk, nuclear risk—they’re all trending to x-risk, and we don’t have control of any of them. They’re all reflections of the same underlying reality: humanity is an infinitely strong infant, with exponentially growing power to imperil itself, but not yet the ability to think or act coherently in response. This is the true threat—we’re in existential danger because our power at scale is growing so much faster than our agency at scale.
This has always been our situation. When we look into the future of AI and see catastrophe, what we’re looking at is not loss of control, but the point at which the rising tide of our power makes our lack of control fatal.
What’s so exciting to me about the cyborgism proposal is that it seems to bear directly on this issue: not just the AI part, all of it. The essence of the current and future LLMs is a collective intelligence they’re learning from our whole species. The nature of the cyborgism proposal is to explore amplified ways of bridging this collective intelligence to individual and collective human agency.
There’s no clear path, but this is the question we need to be asking about simulators and cyborgism. Can they help us scale consciousness and intention the way we’ve already learned to scale power?
The failure modes outlined in the post are daunting, and no doubt there are others. No amount of caution would be too much in pursuing any program involving this type of alien fire. But it’s a mistake to adopt a posture of staying away from the brink—we’re already there.
Thank you for this gorgeously written comment. You really capture the heart of all this so perfectly, and I completely agree with your sentiments.
Yes, I agree that “humanity loses control” has problems, and I would go further. Buddhists claim that the self is an illusion. I don’t know about that, but “humanity” is definitely an illusion if you’re thinking of it as a single agent, similar to a multicellular creature with a central nervous system. So comparing it to an infant doesn’t seem apt. Whatever it is, it’s definitely plural. An ecosystem, maybe?