Peter Watts’ 2006 novel Blindsight has this passage on what it’s like to be a “scrambler”, superintelligent yet nonsentient (in fact superintelligent because it’s unencumbered by sentience), which I read a ~decade ago and found unforgettable:
Imagine you’re a scrambler.
Imagine you have intellect but no insight, agendas but no awareness. Your circuitry hums with strategies for survival and persistence, flexible, intelligent, even technological—but no other circuitry monitors it. You can think of anything, yet are conscious of nothing.
You can’t imagine such a being, can you? The term being doesn’t even seem to apply, in some fundamental way you can’t quite put your finger on.
Try.
Imagine that you encounter a signal. It is structured, and dense with information. It meets all the criteria of an intelligent transmission. Evolution and experience offer a variety of paths to follow, branch-points in the flowcharts that handle such input. Sometimes these signals come from conspecifics who have useful information to share, whose lives you’ll defend according to the rules of kin selection. Sometimes they come from competitors or predators or other inimical entities that must be avoided or destroyed; in those cases, the information may prove of significant tactical value. Some signals may even arise from entities which, while not kin, can still serve as allies or symbionts in mutually beneficial pursuits. You can derive appropriate responses for any of these eventualities, and many others.
You decode the signals, and stumble:
I had a great time. I really enjoyed him. Even if he cost twice as much as any other hooker in the dome—
To fully appreciate Kesey’s Quartet—
They hate us for our freedom—
Pay attention, now—
Understand.
There are no meaningful translations for these terms. They are needlessly recursive. They contain no usable intelligence, yet they are structured intelligently; there is no chance they could have arisen by chance.
The only explanation is that something has coded nonsense in a way that poses as a useful message; only after wasting time and effort does the deception becomes apparent. The signal functions to consume the resources of a recipient for zero payoff and reduced fitness. The signal is a virus.
Viruses do not arise from kin, symbionts, or other allies.
The signal is an attack.
And it’s coming from right about there.
“Now you get it,” Sascha said.
I shook my head, trying to wrap it around that insane, impossible conclusion. “They’re not even hostile.” Not even capable of hostility. Just so profoundly alien that they couldn’t help but treat human language itself as a form of combat.
How do you say We come in peace when the very words are an act of war?
“That’s why they won’t talk to us,” I realized.
“Only if Jukka’s right. He may not be.” It was James again, still quietly resisting, still unwilling to concede a point that even her other selves had accepted. I could see why. Because if Sarasti was right, scramblers were the norm: evolution across the universe was nothing but the endless proliferation of automatic, organized complexity, a vast arid Turing machine full of self-replicating machinery forever unaware of its own existence. And we—we were the flukes and the fossils. We were the flightless birds lauding our own mastery over some remote island while serpents and carnivores washed up on our shores.
Imagine a proliferation of Dyson swarms throughout the cosmos, all computing about as efficiently as physics allows, containing no sentience whatsoever. Bostrom’s Disneyland with no children indeed.
(When I first learned about ChatGPT some years later, my first thought was “they’re eerily reminiscent of scramblers and Rorschach”.)
Why would this be plausible? Watts:
You invest so much in it, don’t you? It’s what elevates you above the beasts of the field, it’s what makes you special. Homo sapiens, you call yourself. Wise Man. Do you even know what it is, this consciousness you cite in your own exaltation? Do you even know what it’s for?
Maybe you think it gives you free will. Maybe you’ve forgotten that sleepwalkers converse, drive vehicles, commit crimes and clean up afterwards, unconscious the whole time. Maybe nobody’s told you that even waking souls are only slaves in denial.
Make a conscious choice. Decide to move your index finger. Too late! The electricity’s already halfway down your arm. Your body began to act a full half-second before your conscious self ‘chose’ to, for the self chose nothing; something else set your body in motion, sent an executive summary—almost an afterthought— to the homunculus behind your eyes. That little man, that arrogant subroutine that thinks of itself as the person, mistakes correlation for causality: it reads the summary and it sees the hand move, and it thinks that one drove the other.
But it’s not in charge. You’re not in charge. If free will even exists, it doesn’t share living space with the likes of you.
Insight, then. Wisdom. The quest for knowledge, the derivation of theorems, science and technology and all those exclusively human pursuits that must surely rest on a conscious foundation. Maybe that’s what sentience would be for— if scientific breakthroughs didn’t spring fully-formed from the subconscious mind, manifest themselves in dreams, as full-blown insights after a deep night’s sleep. It’s the most basic rule of the stymied researcher: stop thinking about the problem. Do something else. It will come to you if you just stop being conscious of it.
Every concert pianist knows that the surest way to ruin a performance is to be aware of what the fingers are doing. Every dancer and acrobat knows enough to let the mind go, let the body run itself. Every driver of any manual vehicle arrives at destinations with no recollection of the stops and turns and roads traveled in getting there. You are all sleepwalkers, whether climbing creative peaks or slogging through some mundane routine for the thousandth time. You are all sleepwalkers.
Don’t even try to talk about the learning curve. Don’t bother citing the months of deliberate practice that precede the unconscious performance, or the years of study and experiment leading up to the gift-wrapped Eureka moment. So what if your lessons are all learned consciously? Do you think that proves there’s no other way? Heuristic software’s been learning from experience for over a hundred years. Machines master chess, cars learn to drive themselves, statistical programs face problems and design the experiments to solve them and you think that the only path to learning leads through sentience? You’re Stone-age nomads, eking out some marginal existence on the veldt—denying even the possibility of agriculture, because hunting and gathering was good enough for your parents.
Do you want to know what consciousness is for? Do you want to know the only real purpose it serves? Training wheels. You can’t see both aspects of the Necker Cube at once, so it lets you focus on one and dismiss the other. That’s a pretty half-assed way to parse reality. You’re always better off looking at more than one side of anything. Go on, try. Defocus. It’s the next logical step.
Oh, but you can’t. There’s something in the way.
And it’s fighting back.
Evolution has no foresight. Complex machinery develops its own agendas. Brains—cheat. Feedback loops evolve to promote stable heartbeats and then stumble upon the temptation of rhythm and music. The rush evoked by fractal imagery, the algorithms used for habitat selection, metastasize into art. Thrills that once had to be earned in increments of fitness can now be had from pointless introspection. Aesthetics rise unbidden from a trillion dopamine receptors, and the system moves beyond modeling the organism. It begins to model the very process of modeling. It consumes ever-more computational resources, bogs itself down with endless recursion and irrelevant simulations. Like the parasitic DNA that accretes in every natural genome, it persists and proliferates and produces nothing but itself. Metaprocesses bloom like cancer, and awaken, and call themselves I.
The system weakens, slows. It takes so much longer now to perceive—to assess the input, mull it over, decide in the manner of cognitive beings. But when the flash flood crosses your path, when the lion leaps at you from the grasses, advanced self-awareness is an unaffordable indulgence. The brain stem does its best. It sees the danger, hijacks the body, reacts a hundred times faster than that fat old man sitting in the CEO’s office upstairs; but every generation it gets harder to work around this— this creaking neurological bureaucracy.
I wastes energy and processing power, self-obsesses to the point of psychosis. Scramblers have no need of it, scramblers are more parsimonious. With simpler biochemistries, with smaller brains—deprived of tools, of their ship, even of parts of their own metabolism—they think rings around you. They hide their language in plain sight, even when you know what they’re saying. They turn your own cognition against itself. They travel between the stars. This is what intelligence can do, unhampered by self-awareness.
Back to scramblers, this time the crew attempting to communicate with them, and the scramblers eventually demonstrating superhuman problem-solving:
This is how you break down the wall:
Start with two beings. They can be human if you like, but that’s hardly a prerequisite. All that matters is that they know how to talk among themselves.
Separate them. Let them see each other, let them speak. Perhaps a window between their cages. Perhaps an audio feed. Let them practice the art of conversation in their own chosen way.
Hurt them.
It may take a while to figure out how. Some may shrink from fire, others from toxic gas or liquid. Some creatures may be invulnerable to blowtorches and grenades, but shriek in terror at the threat of ultrasonic sound. You have to experiment; and when you discover just the right stimulus, the optimum balance between pain and injury, you must inflict it without the remorse.
You leave them an escape hatch, of course. That’s the very point of the exercise: give one of your subjects the means to end the pain, but give the other the information required to use it. To one you might present a single shape, while showing the other a whole selection. The pain will stop when the being with the menu chooses the item its partner has seen. So let the games begin. Watch your subjects squirm. If—when—they trip the off switch, you’ll know at least some of the information they exchanged; and if you record everything that passed between them, you’ll start to get some idea of how they exchanged it.
When they solve one puzzle, give them a new one. Mix things up. Switch their roles. See how they do at circles versus squares. Try them out on factorials and Fibonnaccis. Continue until Rosetta Stone results.
This is how you communicate with a fellow intelligence: you hurt it, and keep on hurting it, until you can distinguish the speech from the screams.
For all his reluctance to accept that these were beings, intelligent and aware, Cunningham had named the prisoners. Stretch tended to float spread-eagled; Clench was the balled-up corner-hugger. … Biotelemetry danced across the headspace beside each alien, luminous annotations shuddering through thin air. I had no idea what constituted normal readings for these creatures, but I couldn’t imagine those jagged spikes passing for anything but bad news. The creatures themselves seethed subtly with fine mosaics in blue and gray, fluid patterns rippling across their cuticles. Perhaps it was a reflexive reaction to the microwaves; for all we knew it was a mating display.
More likely they were screaming.
James killed the microwaves. In the left-hand enclosure, a yellow square dimmed; in the right, an identical icon nested among others had never lit.
The pigment flowed faster in the wake of the onslaught; the arms slowed but didn’t stop. They swept back and forth like listless, skeletal eels.
“Baseline exposure. Five seconds, two hundred fifty Watts.” She spoke for the record. Another affectation; Theseus recorded every breath on board, every trickle of current to five decimal places.
“Repeat,” she said.
The current flickered on, then off.
“Repeat,” she said again.
Not a twitch.
I pointed. “I see it,” she said.
Clench had pressed the tip of one arm against the touchpad. The icon there glowed like a candle flame.
Six and a half minutes later they’d graduated from yellow squares to time-lapsed four-dimensional polyhedrons. It took them as long to distinguish between two twenty-six-faceted shifting solids—differing by one facet in a single frame—as it took them to tell the difference between a yellow square and a red triangle. Intricate patterns played across their surfaces the whole time, dynamic needlepoint mosaics flickering almost too fast to see.
“Fuck,” James whispered.
“Could be splinter skills.” Cunningham had joined us in ConSensus, although his body remained halfway around BioMed.
“Splinter skills,” she repeated dully.
“Savantism. Hyperperformance at one kind of calculation doesn’t necessarily connote high intelligence.”
“I know what splinter skills are, Robert. I just think you’re wrong.”
“Prove it.”
So she gave up on geometry and told the scramblers that one plus one equaled two. Evidently they knew that already: ten minutes later they were predicting ten-digit prime numbers on demand.
She showed them a sequence of two-dimensional shapes; they picked the next one in the series from a menu of subtly-different alternatives. She denied them multiple choice, showed them the beginning of a whole new sequence and taught them to draw on the touch-sensitive interface with the tips of their arms. They finished that series in precise freehand, rendered a chain of logical descendants ending with a figure that led inexorably back to the starting point.
“These aren’t drones.” James’s voice caught in her throat.
“This is all just crunching,” Cunningham said. “Millions of computer programs do it without ever waking up.”
“They’re intelligent, Robert. They’re smarter than us. Maybe they’re smarter than Jukka. And we’re—why can’t you just admit it?”
I could see it all over her: Isaac would have admitted it.
“Because they don’t have the circuitry,” Cunningham insisted. “How could—”
“I don’t know how!” she cried. “That’s your job! All I know is that I’m torturing beings that can think rings around us...”
“Not for much longer, at least. Once you figure out the language—”
She shook her head. “Robert, I haven’t a clue about the language. We’ve been at it for—for hours, haven’t we? The Gang’s all here, language databases four thousand years thick, all the latest linguistic algorithms. And we know exactly what they’re saying, we’re watching every possible way they could be saying it. Right down to the Angstrom.”
“Precisely. So—”
“I’ve got nothing. I know they’re talking through pigment mosaics. There might even be something in the way they move those bristles. But I can’t find the pattern, I can’t even follow how they count, much less tell them I’m...sorry...”
Peter Watts’ 2006 novel Blindsight has this passage on what it’s like to be a “scrambler”, superintelligent yet nonsentient (in fact superintelligent because it’s unencumbered by sentience), which I read a ~decade ago and found unforgettable:
Imagine a proliferation of Dyson swarms throughout the cosmos, all computing about as efficiently as physics allows, containing no sentience whatsoever. Bostrom’s Disneyland with no children indeed.
(When I first learned about ChatGPT some years later, my first thought was “they’re eerily reminiscent of scramblers and Rorschach”.)
Why would this be plausible? Watts:
Back to scramblers, this time the crew attempting to communicate with them, and the scramblers eventually demonstrating superhuman problem-solving:
It’s very funny that Rorschach linguistic ability is totally unremarkable comparing to modern LLMs.