Sensual Experience

Modern day gamemak­ers are con­stantly work­ing on higher-re­s­olu­tion, more re­al­is­tic graph­ics; more im­mer­sive sounds—but they’re a long long way off real life.

Press­ing the “W” key to run for­ward as a graphic of a hun­gry tiger bounds be­hind you, just doesn’t seem quite as sen­sual as run­ning fran­ti­cally across the sa­vanna with your own legs, breath­ing in huge gasps and pump­ing your arms as the sun beats down on your shoulders, the grass brushes your shins, and the air whips around you with the wind of your pas­sage.

Don’t mis­take me for a lud­dite; I’m not say­ing the tech­nol­ogy can’t get that good. I’m say­ing it hasn’t got­ten that good yet.

Failing to es­cape the com­puter tiger would also have fewer long-term con­se­quences than failing to es­cape a biolog­i­cal tiger—it would be less a part of the to­tal story of your life—mean­ing you’re also likely to be less emo­tion­ally in­volved. But that’s a topic for an­other post. To­day’s post is just about the sen­sual qual­ity of the ex­pe­rience.

Sen­sual ex­pe­rience isn’t a ques­tion of some mys­te­ri­ous qual­ity that only the “real world” pos­sesses. A com­puter screen is as real as a tiger, af­ter all. What­ever is, is real.

But the pat­tern of the pseudo-tiger, in­side the com­puter chip, is nowhere near as com­plex as a biolog­i­cal tiger; it offers far fewer modes in which to in­ter­act. And the sen­sory band­width be­tween you and the com­puter’s pseudo-world is rel­a­tively low; and the in­for­ma­tion pass­ing along it isn’t in quite the right for­mat.

It’s not a ques­tion of com­puter tigers be­ing “vir­tual” or “simu­lated”, and there­fore some­how a sep­a­rate mag­is­terium. But with pre­sent tech­nol­ogy, and the way your brain is presently set up, you’d have a lot more neu­rons in­volved in run­ning away from a biolog­i­cal tiger.

Run­ning would fill your whole vi­sion with mo­tion, not just a flat rec­t­an­gu­lar screen—which trans­lates into more square cen­time­ters of vi­sual cor­tex get­ting ac­tively en­gaged.

The graph­ics on a com­puter mon­i­tor try to trig­ger your sense of spa­tial mo­tion (re­sid­ing in the pari­etal cor­tex, btw). But they’re pre­sent­ing the in­for­ma­tion differ­ently from its na­tive for­mat —with­out binoc­u­lar vi­sion, for ex­am­ple, and with­out your vestibu­lar senses in­di­cat­ing true mo­tion. So the sense of mo­tion isn’t likely to be quite the same, what it would be if you were run­ning.

And there’s the sense of touch that in­di­cates the wind on your skin; and the pro­pri­o­cep­tive sen­sors that re­spond to the po­si­tion of your limbs; and the nerves that record the strain on your mus­cles. There’s a whole strip of sen­so­ri­mo­tor cor­tex run­ning along the top of your brain, that would be much more in­tensely in­volved in “real” run­ning.

It’s a very old ob­ser­va­tion, that Homo sapi­ens was made to hunt and gather on the sa­vanna, rather than work in an office. Civ­i­liza­tion and its dis­con­tents… But aliena­tion needs a causal mechanism; it doesn’t just hap­pen by magic. Physics is physics, so it’s not that one en­vi­ron­ment is less real than an­other. But our brains are more adapted to in­ter­fac­ing with jun­gles than com­puter code.

Writ­ing a com­pli­cated com­puter pro­gram car­ries its own triumphs and failures, heights of ex­ul­ta­tion and pits of de­spair. But is it the same sort of sen­sual ex­pe­rience as, say, rid­ing a mo­tor­cy­cle? I’ve never ac­tu­ally rid­den a mo­tor­cy­cle, but I ex­pect not.

I’ve ex­pe­rienced the ex­hil­a­ra­tion of get­ting a pro­gram right on the dozenth try af­ter fi­nally spot­ting the prob­lem. I doubt a ran­dom mo­ment of a mo­tor­cy­cle ride ac­tu­ally feels bet­ter than that. But still, my hunter-gath­erer an­ces­tors never wrote com­puter pro­grams. And so my mind’s grasp on code is main­tained us­ing more rar­efied, more ab­stract, more gen­eral ca­pa­bil­ities—which means less sen­sual in­volve­ment.

Doesn’t com­puter pro­gram­ming de­serve to be as much of a sen­sual ex­pe­rience as mo­tor­cy­cle rid­ing? Some time ago, a rel­a­tive once asked me if I thought that com­puter pro­gram­ming could use all my tal­ents; I at once replied, “There is no limit to the tal­ent you can use in com­puter pro­gram­ming.” It’s as close as hu­man be­ings have ever come to play­ing with the raw stuff of cre­ation—but our grasp on it is too dis­tant from the jun­gle. All our in­volve­ment is through let­ters on a com­puter screen. I win, and I’m happy, but there’s no wind on my face.

If only my an­ces­tors back to the level of my last com­mon an­ces­tor with a mouse, had con­stantly faced the challenge of writ­ing com­puter pro­grams! Then I would have brain ar­eas suited to the task, and pro­gram­ming com­put­ers would be more of a sen­sual ex­pe­rience...

Per­haps it’s not too late to fix the mis­take?

If there were some­thing around that was smart enough to rewrite hu­man brains with­out break­ing them—not a triv­ial amount of smart­ness—then it would be pos­si­ble to ex­pand the range of things that are sen­su­ally fun.

Not just novel challenges, but novel high-band­width senses and cor­re­spond­ing new brain ar­eas. Wi­den­ing the sen­so­rium to in­clude new vivid, de­tailed ex­pe­riences. And not ne­glect­ing the other half of the equa­tion, high-band­width mo­tor con­nec­tions—new mo­tor brain ar­eas, to con­trol with sub­tlety our new limbs (the parts of the pro­cess that we con­trol as our di­rect han­dles on it).

There’s a story—old now, but I re­mem­ber how ex­cit­ing it was when the news first came out—about a brain-com­puter in­ter­face for a “locked-in” pa­tient (who could pre­vi­ously only move his eyes), con­nect­ing con­trol of a com­puter cur­sor di­rectly to neu­rons in his vi­sual cor­tex. It took some train­ing at first for him to use the cur­sor—he started out by try­ing to move his par­a­lyzed arm, which was the part of the mo­tor cor­tex they were in­ter­fac­ing, and watched as the cur­sor jerked around on the screen. But af­ter a while, they asked the pa­tient, “What does it feel like?” and the pa­tient replied, “It doesn’t feel like any­thing.” He just con­trol­led the cur­sor the same sort of way he would have con­trol­led a finger, ex­cept that it wasn’t a finger, it was a cur­sor.

Like most brain mod­ifi­ca­tions, adding new senses is not some­thing to be done lightly. Sen­sual ex­pe­rience too eas­ily ren­ders a task in­volv­ing.

Con­sider taste buds. Rec­og­niz­ing the taste of the same food on differ­ent oc­ca­sions was very im­por­tant to our an­ces­tors—it was how they learned what to eat, that ex­tracted reg­u­lar­ity. And our an­ces­tors also got helpful re­in­force­ment from their taste buds about what to eat—re­in­force­ment which is now worse than use­less, be­cause of the mar­ket­ing in­cen­tive to re­verse-en­g­ineer tasti­ness us­ing ar­tifi­cial sub­stances. By now, it’s prob­a­bly true that at least some peo­ple have eaten 162,329 potato chips in their life­times. That’s even less nov­elty and challenge than carv­ing 162,329 table legs.

I’m not say­ing we should try to elimi­nate our senses of taste. There’s a lot to be said for grand­father­ing in the senses we started with—it pre­serves our ex­ist­ing life mem­o­ries, for ex­am­ple. Once you re­al­ize how easy it would be for a mind to col­lapse into a plea­sure cen­ter, you start to re­spect the “com­pli­ca­tions” of your goal sys­tem a lot more, and be more wary around “sim­plifi­ca­tions”.

But I do want to nudge peo­ple into adopt­ing some­thing of a ques­tion­ing at­ti­tude to­ward the senses we have now, rather than as­sum­ing that the ex­ist­ing senses are The Way Things Have Been And Will Always Be. A sex or­gan bears thou­sands of densely packed nerves for sig­nal strength, but that sig­nal—how­ever strong—isn’t as com­pli­cated as the sen­sa­tions sent out by taste buds. Is that re­ally ap­pro­pri­ate for one of the most in­ter­est­ing parts of hu­man ex­is­tence? That even a novice chef can cre­ate a wider va­ri­ety of taste sen­sa­tions for your tongue, than—well, I’d bet­ter stop there. But from a fun-the­o­retic stand­point, the ex­ist­ing setup is wildly un­bal­anced in a lot of ways. It wasn’t de­signed for the sake of eu­daimo­nia.

I con­clude with the fol­low­ing cau­tion­ary quote from an old IRC con­ver­sa­tion, as a re­minder that maybe not ev­ery­thing should be a sen­sual ex­pe­rience:

<MRAmes> I want a sen­sory modal­ity for reg­u­lar ex­pres­sions.

Part of The Fun The­ory Sequence

Next post: “Liv­ing By Your Own Strength

Pre­vi­ous post: “Con­tin­u­ous Im­prove­ment