Living By Your Own Strength

Fol­lowup to: Truly Part of You

“My­self, and Morisato-san… we want to live to­gether by our own strength.”

Jared Di­a­mond once called agri­cul­ture “the worst mis­take in the his­tory of the hu­man race”. Farm­ers could grow more wheat than hunter-gath­er­ers could col­lect nuts, but the ev­i­dence seems pretty con­clu­sive that agri­cul­ture traded qual­ity of life for quan­tity of life. One study showed that the farm­ers in an area were six inches shorter and seven years shorter-lived than their hunter-gath­erer pre­de­ces­sors—even though the farm­ers were more nu­mer­ous.

I don’t know if I’d call agri­cul­ture a mis­take. But one should at least be aware of the down­sides. Policy de­bates should not ap­pear one-sided.

In the same spirit—

Once upon a time, our hunter-gath­erer an­ces­tors strung their own bows, wove their own bas­kets, whit­tled their own flutes.

And part of our aliena­tion from that en­vi­ron­ment of evolu­tion­ary adapt­ed­ness, is the num­ber of tools we use that we don’t un­der­stand and couldn’t make for our­selves.

You can look back on Over­com­ing Bias, and see that I’ve always been sus­pi­cious of bor­rowed strength. (Even be­fore I un­der­stood the source of Robin’s and my dis­agree­ment about the Sin­gu­lar­ity, that is.) In Guess­ing the Teacher’s Pass­word I talked about the (well-known) prob­lem in which schools end up teach­ing ver­bal be­hav­ior rather than real knowl­edge. In Truly Part of You I sug­gested one test for false knowl­edge: Imag­ine delet­ing a fact from your mind, and ask if it would grow back.

I know many ways to prove the Pythagorean The­o­rem, in­clud­ing at least one proof that is purely vi­sual and can be seen at a glance. But if you deleted the Pythagorean The­o­rem from my mind en­tirely, would I have enough math skills left to grow it back the next time I needed it? I hope so—cer­tainly I’ve solved math prob­lems that seem tougher than that, what with benefit of hind­sight and all. But, as I’m not an AI, I can’t ac­tu­ally switch off the mem­o­ries and as­so­ci­a­tions, and test my­self in that way.

Wield­ing some­one else’s strength to do things be­yond your own un­der­stand­ing—that re­ally is as dan­ger­ous as the Deeply Wise phras­ing makes it sound.

I ob­served in Failing to Learn from His­tory (mus­ing on my child­hood fool­ish­ness in offer­ing a mys­te­ri­ous an­swer to a mys­te­ri­ous ques­tion): “If only I had per­son­ally pos­tu­lated as­trolog­i­cal mys­ter­ies and then dis­cov­ered New­to­nian me­chan­ics, pos­tu­lated al­chem­i­cal mys­ter­ies and then dis­cov­ered chem­istry, pos­tu­lated vi­tal­is­tic mys­ter­ies and then dis­cov­ered biol­ogy. I would have thought of my Mys­te­ri­ous An­swer and said to my­self: No way am I fal­ling for that again.

At that point in my child­hood, I’d been handed some tech­niques of ra­tio­nal­ity but I didn’t ex­actly own them. Bor­row­ing some­one else’s knowl­edge re­ally doesn’t give you any­thing re­motely like the same power level re­quired to dis­cover that knowl­edge for your­self.

Would Isaac New­ton have re­mained a mys­tic, even in that ear­lier era, if he’d lived the lives of Gal­ileo and Archimedes in­stead of just read­ing about them? If he’d per­son­ally seen the planets re­duced from gods to spheres in a telescope? If he’d per­son­ally fought that whole war against ig­no­rance and mys­tery that had to be fought, be­fore Isaac New­ton could be handed math and sci­ence as a start to his fur­ther work?

We stand on the shoulders of gi­ants, and in do­ing so, the power that we wield is far out of pro­por­tion to the power that we could gen­er­ate for our­selves. This is true even of our rev­olu­tion­ar­ies. And yes, we couldn’t be­gin to sup­port this world if peo­ple could only use their own strength. Even so, we are los­ing some­thing.

That thought oc­curred to me, read­ing about the Man­hat­tan Pro­ject, and the pe­ti­tion that the physi­cists signed to avoid drop­ping the atomic bomb on Ja­pan. It was too late, of course; they’d already built the bombs and handed them to the mil­i­tary, and they couldn’t take back that gift. And so nu­clear weapons passed into the hands of poli­ti­ci­ans, who could never have cre­ated such a thing through their own strength...

Not that I’m say­ing the world would nec­es­sar­ily have been a bet­ter place, if physi­cists had pos­sessed sole cus­tody of ICBMs. What does a physi­cist know about in­ter­na­tional diplo­macy, or war? And it’s not as if Leo Szilard—who first thought of the fis­sion chain re­ac­tion—had per­son­ally in­vented sci­ence; he too was us­ing pow­ers be­yond his own strength. And it’s not as if the physi­cists on the Man­hat­tan Pro­ject raised the money to pay for their salaries and their ma­te­ri­als; they were bor­row­ing the strength of the poli­ti­ci­ans...

But if no one had been able to use nu­clear weapons with­out, say, pos­sess­ing the dis­ci­pline of a sci­en­tist and the dis­ci­pline of a poli­ti­cian—with­out per­son­ally know­ing enough to con­struct an atomic bomb and make friends—the world might have been a slightly safer place.

And if no­body had been able to con­struct an atomic bomb with­out first dis­cov­er­ing for them­selves the na­ture and ex­is­tence of physics, then we would have been much safer from atomic bombs, be­cause no one would have been able to build them un­til they were two hun­dred years old.

With hu­mans leav­ing the game af­ter just sev­enty years, we couldn’t sup­port this world us­ing only our own strengths. But we have traded qual­ity of in­sight for quan­tity of in­sight.

It does some­times seem to me that many of this world’s prob­lems, stem from our us­ing pow­ers that aren’t ap­pro­pri­ate to sev­enty-year-olds.

And there is a higher level of strength-own­er­ship, which no hu­man be­ing has yet achieved. Even when we run, we’re just us­ing the mus­cles that evolu­tion built for us. Even when we think, we’re just us­ing the brains that evolu­tion built for us.

I’m not sug­gest­ing that peo­ple should cre­ate them­selves from scratch with­out a start­ing point. Just point­ing out that it would be a differ­ent world if we un­der­stood our own brains and could re­design our own legs. As yet there’s no hu­man “ra­tio­nal­ist” or “sci­en­tist”, what­ever they know about “how to think”, who could ac­tu­ally build a ra­tio­nal AI—which shows you the limits of our self-un­der­stand­ing.

This is not the sort of thing that I’d sug­gest as an im­me­di­ate al­ter­a­tion. I’m not sug­gest­ing that peo­ple should in­stantly on a silver plat­ter be given full knowl­edge of how their own brains work and the abil­ity to re­design their own legs. Be­cause maybe peo­ple will be bet­ter off if they aren’t given that kind of power, but rather have to work out the an­swer for them­selves.

Just in terms of anomie ver­sus fun, there’s a big differ­ence be­tween be­ing able to do things for your­self, and hav­ing to rely on other peo­ple to do them for you. (Even if you’re do­ing them with a brain you never de­signed your­self.)

I don’t know if it’s a prin­ci­ple that would stay un­til the end of time, to the chil­dren’s chil­dren. Maybe bet­ter-de­signed minds could han­dle opaque tools with­out the anomie.

But it is part of the com­monly re­told prophecy of Ar­tifi­cial In­tel­li­gence and the Age of the Ma­chine, that this era must be ac­com­panied by greater re­li­ance on things out­side your­self, more in­com­pre­hen­si­ble tools into which you have less in­sight and less part in their cre­ation.

Such a prophecy is not sur­pris­ing. That is the way the trend has gone so far, in our cul­ture that is too busy stay­ing al­ive to op­ti­mize for fun. From the fire-start­ing tools that you built your­self, to the village can­dle­sel­ler, and then from the can­dle­sel­ler to the elec­tric light that runs on strange math­e­mat­i­cal prin­ci­ples and is pow­ered by a dis­tant gen­er­a­tor… we are sur­rounded by things out­side our­selves and strengths out­side our un­der­stand­ing; we need them to stay al­ive, or we buy them be­cause it’s eas­ier that way.

But with a suffi­cient sur­plus of power, you could start do­ing things the eu­daimonic way. Start re­think­ing the life ex­pe­rience as a road to in­ter­nal­iz­ing new strengths, in­stead of just try­ing to keep peo­ple al­ive effi­ciently.

A Friendly AI doesn’t have to be a con­tinu­a­tion of ex­ist­ing trends. It’s not the Ma­chine. It’s not the alien force of tech­nol­ogy. It’s not mech­a­niz­ing a fac­tory. It’s not a new gad­get for sale. That’s not where the shape comes from. What it is—is not easy to ex­plain; but I’m re­minded of doc Smith’s de­scrip­tion of the Lens as “the phys­i­cal man­i­fes­ta­tion of a purely philo­soph­i­cal con­cept”. That philo­soph­i­cal con­cept doesn’t have to man­i­fest as new but­tons to press—if, on re­flec­tion, that’s not what we would want.

Part of The Fun The­ory Sequence

Next post: “Free to Op­ti­mize

Pre­vi­ous post: “Sen­sual Ex­pe­rience