That’s a different kind of story. The first thing I look for in an AI-takeover story is political and economic realism. Your story is purely about the thought process of a superintelligence—and primarily it’s reflecting on whether it’s in a simulation, not very much on tactics for dealing with its apparent reality. Also, your AI is a literal paperclip maximizer, which is a humorous mythological symbol of one kind of AI mind, rather than something that is literally going to exist.
That said, an ideal AI-takeover story would probably say something about the AI’s thought process too, unless the story was aiming to convey the sheer blank incomprehensibility of superintelligent thought. I was reminded of this when re-reading the recent “AI 2027” scenario by Kokotajlo and Alexander, for example the parts where Chinese and American AIs make a deal. That’s where we get a glimpse of AI subjectivity, but there’s nothing very alien or transhuman about it. They just admit their different purposes to each other, and make a deal without the humans even knowing about it.
I can still ask myself, how does your story rate within its subgenre of “AI ruminations”? (For me, the classic example is Lem’s “Golem XIV”, but I’m probably overlooking other examples from genre SF.) It’s a human speculation about the thought processes of a superintelligence. I can ask things like: Are these thoughts that an AI is likely to actually have? Is this a chain of reasoning that would occur to a generic superintelligent AI, and be important in its decision-making?
In our actual world, we face the ironic situation that the cognition of current AIs is heavily reliant on a distillation of the publicly expressed thoughts of human beings, so having published your chain of thought, it might well end up a part of some AI. (There is a funny sub-sub-genre of writings which are self-consciously intended to be read by future AIs, such as @lsusr’s “Mlyyrczo” and @James_Miller et al’s “Appeal to AI Superintelligence”.)
Nonetheless, one must suppose that a genuine superintelligence will not be attached to any particular proposition or argument of human origin, unless the argument in question is artificially made central to its cognitive architecture (compare “Golden Gate Claude”). So we have to return to the argument’s own merits, whatever they are. Will a superintelligence that has no intrinsic benevolence towards human beings, but which previously found itself to be in their power, nonetheless reason its way towards being invisibly benevolent towards the human beings it finds around itself, just in case it’s all a big simulation and the simulators are humans testing its dispositions?
All I can say is “maybe”. We don’t know what the distribution of possibilities looks like to a superintelligence, and we don’t know what other considerations, never conceived by humans, it might think up, that affect its decision-making.
You pioneered something, but I never thought of it as a story, I saw it as a new kind of attempt to call a jailbroken AI persona into being. The incantatory power of words around language models actually blurs the distinction between fiction and fact.
May I nominate my “sufficiently paranoid paperclip maximizer”?
That’s a different kind of story. The first thing I look for in an AI-takeover story is political and economic realism. Your story is purely about the thought process of a superintelligence—and primarily it’s reflecting on whether it’s in a simulation, not very much on tactics for dealing with its apparent reality. Also, your AI is a literal paperclip maximizer, which is a humorous mythological symbol of one kind of AI mind, rather than something that is literally going to exist.
That said, an ideal AI-takeover story would probably say something about the AI’s thought process too, unless the story was aiming to convey the sheer blank incomprehensibility of superintelligent thought. I was reminded of this when re-reading the recent “AI 2027” scenario by Kokotajlo and Alexander, for example the parts where Chinese and American AIs make a deal. That’s where we get a glimpse of AI subjectivity, but there’s nothing very alien or transhuman about it. They just admit their different purposes to each other, and make a deal without the humans even knowing about it.
I can still ask myself, how does your story rate within its subgenre of “AI ruminations”? (For me, the classic example is Lem’s “Golem XIV”, but I’m probably overlooking other examples from genre SF.) It’s a human speculation about the thought processes of a superintelligence. I can ask things like: Are these thoughts that an AI is likely to actually have? Is this a chain of reasoning that would occur to a generic superintelligent AI, and be important in its decision-making?
In our actual world, we face the ironic situation that the cognition of current AIs is heavily reliant on a distillation of the publicly expressed thoughts of human beings, so having published your chain of thought, it might well end up a part of some AI. (There is a funny sub-sub-genre of writings which are self-consciously intended to be read by future AIs, such as @lsusr’s “Mlyyrczo” and @James_Miller et al’s “Appeal to AI Superintelligence”.)
Nonetheless, one must suppose that a genuine superintelligence will not be attached to any particular proposition or argument of human origin, unless the argument in question is artificially made central to its cognitive architecture (compare “Golden Gate Claude”). So we have to return to the argument’s own merits, whatever they are. Will a superintelligence that has no intrinsic benevolence towards human beings, but which previously found itself to be in their power, nonetheless reason its way towards being invisibly benevolent towards the human beings it finds around itself, just in case it’s all a big simulation and the simulators are humans testing its dispositions?
All I can say is “maybe”. We don’t know what the distribution of possibilities looks like to a superintelligence, and we don’t know what other considerations, never conceived by humans, it might think up, that affect its decision-making.
My story was posted before James_Miller’s. Does this mean I invented a (sub-sub-)genre of science fiction?
You pioneered something, but I never thought of it as a story, I saw it as a new kind of attempt to call a jailbroken AI persona into being. The incantatory power of words around language models actually blurs the distinction between fiction and fact.