In Peter Watts’ Blindsight, Siri Keeton explains what he is:
This is what I am:
I am the bridge between the bleeding edge and the dead center. I stand between the Wizard of Oz and the man behind the curtain.
I am the curtain.
I am not an entirely new breed. My roots reach back to the dawn of civilization but those precursors served a different function, a less honorable one. They only greased the wheels of social stability; they would sugarcoat unpleasant truths, or inflate imaginary bogeymen for political expedience. They were vital enough in their way. Not even the most heavily-armed police state can exert brute force on all of its citizens all of the time. Meme management is so much subtler; the rose-tinted refraction of perceived reality, the contagious fear of threatening alternatives. There have always been those tasked with the rotation of informational topologies, but throughout most of history they had little to do with increasing its clarity.
The new Millennium changed all that. We’ve surpassed ourselves now, we’re exploring terrain beyond the limits of merely human understanding. Sometimes its contours, even in conventional space, are just too intricate for our brains to track; other times its very axes extend into dimensions inconceivable to minds built to fuck and fight on some prehistoric grassland. So many things constrain us, from so many directions. The most altruistic and sustainable philosophies fail before the brute brain-stem imperative of self-interest. Subtle and elegant equations predict the behavior of the quantum world, but none can explain it. After four thousand years we can’t even prove that reality exists beyond the mind of the first-person dreamer. We have such need of intellects greater than our own.
But we’re not very good at building them. The forced matings of minds and electrons succeed and fail with equal spectacle. Our hybrids become as brilliant as savants, and as autistic. We graft people to prosthetics, make their overloaded motor strips juggle meat and machinery, and shake our heads when their fingers twitch and their tongues stutter. Computers bootstrap their own offspring, grow so wise and incomprehensible that their communiqués assume the hallmarks of dementia: unfocused and irrelevant to the barely-intelligent creatures left behind.
And when your surpassing creations find the answers you asked for, you can’t understand their analysis and you can’t verify their answers. You have to take their word on faith—
—Or you use information theory to flatten it for you, to squash the tesseract into two dimensions and the Klein bottle into three, to simplify reality and pray to whatever Gods survived the millennium that your honorable twisting of the truth hasn’t ruptured any of its load-bearing pylons. You hire people like me; the crossbred progeny of profilers and proof assistants and information theorists.
While the technicalities don’t make much sense, spiritually I related to Siri’s self-description a lot when I first read it over a decade ago, in that I was recognised as very good at a particular kind of distillation (in straightforwardly verifiable domains) well beyond my actual understanding of the material, the latter being verifiable because I’d sometimes say something anyone who’d grokked the topic would trivially recognise as nonsense, which made me feel like my thinking was much more “structural/syntactic” than “semantic/gearsy”.
Spiritually, frontier models feel like my brain on steroids. Experiencing them surpass me at the thing I was rewarded for being good at in my youth has been interesting.
I had a classmate at university, who could solve various mathematical problems and then say “I have actually no idea what any of this means, I just wrote the symbols in a way that feels correct” and he was correct quite often (but not always) that we copied his homework when we didn’t understand what it means, because it was more reliable than making our own attempt.
In Peter Watts’ Blindsight, Siri Keeton explains what he is:
While the technicalities don’t make much sense, spiritually I related to Siri’s self-description a lot when I first read it over a decade ago, in that I was recognised as very good at a particular kind of distillation (in straightforwardly verifiable domains) well beyond my actual understanding of the material, the latter being verifiable because I’d sometimes say something anyone who’d grokked the topic would trivially recognise as nonsense, which made me feel like my thinking was much more “structural/syntactic” than “semantic/gearsy”.
Spiritually, frontier models feel like my brain on steroids. Experiencing them surpass me at the thing I was rewarded for being good at in my youth has been interesting.
I had a classmate at university, who could solve various mathematical problems and then say “I have actually no idea what any of this means, I just wrote the symbols in a way that feels correct” and he was correct quite often (but not always) that we copied his homework when we didn’t understand what it means, because it was more reliable than making our own attempt.