Thank you for this amazing and clarifying post.
You’re operating far above my pay grade in connection with any of this subject matter, but nonetheless I’m going to dare a different suggestion for the True Names: do you think there’s any merit to -speculators- and -speculations-? I believe these names fit all the excellent and clarifying tests and criteria presented in your post; in particular those referencing counterfactual configurations and process specification through chaining. Furthermore I think they have some advantages of their own. Speculators producing speculations seem more the right relationship between the two main concepts than simulators producing simulacra. (I don’t think they do that!) Also, simulators have such a long history in digital systems of being aimed at deterministic fidelity to a reference system, which could be at odds with the abundant production of counterfactuals I believe you’re actually seeking to emphasize here. Finally, speculations can be fanciful, realistic, or absurd, a nice match to the variety of outputs produced by GPT in the presence of different types of prompting, something you highlight, I think correctly, as a hallmark of GPT’s status as a novel type of AI. One who speculates is a certain type of thinker: I propose that GPT is that type.
What do you think?
This is a beautiful and clarifying post, which I found just as thrilling to read as I did janus’s original Simulators post—a high bar. Thank you!
Many comments come to mind. I’ll start with one around the third core claim in the Introduction: “Unless we manage to coordinate around it, the default outcome is that humanity will eventually be disempowered by a powerful autonomous agent (or agents).” The accompanying graph shows us a point an unknown distance into the future where “Humanity loses control”.
The urgency is correct, but this isn’t the right threat. All three words are wrong: control is too blunt an instrument, you can’t lose what you never had, and humanity has no referent capable of carrying the load we’d like to put on it here.
Humanity doesn’t have control of even today’s AI, but it’s not just AI: climate risk, pandemic risk, geopolitical risk, nuclear risk—they’re all trending to x-risk, and we don’t have control of any of them. They’re all reflections of the same underlying reality: humanity is an infinitely strong infant, with exponentially growing power to imperil itself, but not yet the ability to think or act coherently in response. This is the true threat—we’re in existential danger because our power at scale is growing so much faster than our agency at scale.
This has always been our situation. When we look into the future of AI and see catastrophe, what we’re looking at is not loss of control, but the point at which the rising tide of our power makes our lack of control fatal.
What’s so exciting to me about the cyborgism proposal is that it seems to bear directly on this issue: not just the AI part, all of it. The essence of the current and future LLMs is a collective intelligence they’re learning from our whole species. The nature of the cyborgism proposal is to explore amplified ways of bridging this collective intelligence to individual and collective human agency.
There’s no clear path, but this is the question we need to be asking about simulators and cyborgism. Can they help us scale consciousness and intention the way we’ve already learned to scale power?
The failure modes outlined in the post are daunting, and no doubt there are others. No amount of caution would be too much in pursuing any program involving this type of alien fire. But it’s a mistake to adopt a posture of staying away from the brink—we’re already there.