On Do­ing the Impossible

“Per­severe.” It’s a piece of ad­vice you’ll get from a whole lot of high achiev­ers in a whole lot of dis­cip­lines. I didn’t un­der­stand it at all, at first.

At first, I thought “per­sever­ance” meant work­ing 14-hour days. Ap­par­ently, there are people out there who can work for 10 hours at a tech­nical job, and then, in their mo­ments between eat­ing and sleep­ing and go­ing to the bath­room, seize that un­filled spare time to work on a book. I am not one of those people—it still hurts my pride even now to con­fess that. I’m work­ing on some­thing im­port­ant; shouldn’t my brain be will­ing to put in 14 hours a day? But it’s not. When it gets too hard to keep work­ing, I stop and go read or watch some­thing. Be­cause of that, I thought for years that I en­tirely lacked the vir­tue of “per­sever­ance”.

In ac­cord­ance with hu­man nature, Eliezer1998 would think things like: “What counts is out­put, not in­put.” Or, “Lazi­ness is also a vir­tue—it leads us to back off from fail­ing meth­ods and think of bet­ter ways.” Or, “I’m do­ing bet­ter than other people who are work­ing more hours. Maybe, for cre­at­ive work, your mo­ment­ary peak out­put is more im­port­ant than work­ing 16 hours a day.” Per­haps the fam­ous sci­ent­ists were se­duced by the Deep Wis­dom of say­ing that “hard work is a vir­tue”, be­cause it would be too aw­ful if that coun­ted for less than in­tel­li­gence?

I didn’t un­der­stand the vir­tue of per­sever­ance un­til I looked back on my jour­ney through AI, and real­ized that I had over­es­tim­ated the dif­fi­culty of al­most every single im­port­ant prob­lem.

Sounds crazy, right? But bear with me here.

When I was first de­cid­ing to chal­lenge AI, I thought in terms of 40-year times­cales, Man­hat­tan Pro­jects, plan­et­ary com­put­ing net­works, mil­lions of pro­gram­mers, and pos­sibly aug­men­ted hu­mans.

This is a com­mon fail­ure mode in AI-fu­tur­ism which I may write about later; it con­sists of the leap from “I don’t know how to solve this” to “I’ll ima­gine throw­ing some­thing really big at it”. So­mething huge enough that, when you ima­gine it, that ima­gin­a­tion cre­ates a feel­ing of im­press­ive­ness strong enough to be com­men­sur­able with the prob­lem. (There’s a fel­low cur­rently on the AI list who goes around say­ing that AI will cost a quad­ril­lion dol­lars—we can’t get AI without spend­ing a quad­ril­lion dol­lars, but we could get AI at any time by spend­ing a quad­ril­lion dol­lars.) This, in turn, lets you ima­gine that you know how to solve AI, without try­ing to fill the ob­vi­ously-im­possible de­mand that you un­der­stand in­tel­li­gence.

So, in the be­gin­ning, I made the same mis­take: I didn’t un­der­stand in­tel­li­gence, so I ima­gined throw­ing a Man­hat­tan Pro­ject at the prob­lem.

But, hav­ing cal­cu­lated the plan­et­ary death rate at 55 mil­lion per year or 150,000 per day, I did not turn around and run away from the big scary prob­lem like a frightened rab­bit. In­stead, I star­ted try­ing to fig­ure out what kind of AI pro­ject could get there fast­est. If I could make the Sin­gu­lar­ity hap­pen one hour earlier, that was a reas­on­able re­turn on in­vest­ment for a pre-Sin­gu­lar­ity ca­reer. (I wasn’t think­ing in terms of ex­ist­en­tial risks or Friendly AI at this point.)

So I didn’t run away from the big scary prob­lem like a frightened rab­bit, but stayed to see if there was any­thing I could do.

Fun his­tor­ical fact: In 1998, I’d writ­ten this long treat­ise pro­pos­ing how to go about cre­at­ing a self-im­prov­ing or “seed” AI (a term I had the honor of coin­ing). Brian Atkins, who would later be­come the found­ing fun­der of the Sin­gu­lar­ity In­sti­tute, had just sold Hyper­mart to Go2Net. Brian emailed me to ask whether this AI pro­ject I was de­scrib­ing was some­thing that a reas­on­able-sized team could go out and ac­tu­ally do. “No,” I said, “it would take a Man­hat­tan Pro­ject and thirty years,” so for a while we were con­sid­er­ing a new dot-com star­tup in­stead, to cre­ate the fund­ing to get real work done on AI...

A year or two later, after I’d heard about this new­fangled “open source” thing, it seemed to me that there was some pre­lim­in­ary de­vel­op­ment work—new com­puter lan­guages and so on—that a small or­gan­iz­a­tion could do; and that was how the Sin­gu­lar­ity In­sti­tute star­ted.

This strategy was, of course, en­tirely wrong.

But even so, I went from “There’s noth­ing I can do about it now” to “Hm… maybe there’s an in­cre­mental path through open-source de­vel­op­ment, if the ini­tial ver­sions are use­ful to enough people.”

This is back at the dawn of time, so I’m not say­ing any of this was a good idea. But in terms of what I thought I was try­ing to do, a year of cre­at­ive think­ing had shortened the ap­par­ent path­way: The prob­lem looked slightly less im­possible than it did the very first time I ap­proached it.

The more in­ter­est­ing pat­tern is my entry into Friendly AI. Ini­tially, Friendly AI hadn’t been some­thing that I had con­sidered at all—be­cause it was ob­vi­ously im­possible and use­less to de­ceive a su­per­in­tel­li­gence about what was the right course of ac­tion.

So, his­tor­ic­ally, I went from com­pletely ig­nor­ing a prob­lem that was “im­possible”, to tak­ing on a prob­lem that was merely ex­tremely dif­fi­cult.

Nat­ur­ally this in­creased my total work­load.

Same thing with try­ing to un­der­stand in­tel­li­gence on a pre­cise level. Ori­gin­ally, I’d writ­ten off this prob­lem as im­possible, thus re­mov­ing it from my work­load. (This lo­gic seems pretty de­ranged in ret­ro­spect—Nature doesn’t care what you can’t do when It’s writ­ing your pro­ject re­quire­ments—but I still see AI­folk try­ing it all the time.) To hold my­self to a pre­cise stand­ard meant put­ting in more work than I’d pre­vi­ously ima­gined I needed. But it also meant tack­ling a prob­lem that I would have dis­missed as en­tirely im­possible not too much earlier.

Even though in­di­vidual prob­lems in AI have seemed to be­come less in­tim­id­at­ing over time, the total moun­tain-to-be-climbed has in­creased in height—just like con­ven­tional wis­dom says is sup­posed to hap­pen—as prob­lems got taken off the “im­possible” list and put on the “to do” list.

I star­ted to un­der­stand what was hap­pen­ing—and what “Per­severe!” really meant—at the point where I no­ticed other AI­folk do­ing the same thing: say­ing “Im­possible!” on prob­lems that seemed em­in­ently solv­able—re­l­at­ively more straight­for­ward, as such things go. But they were things that would have seemed vastly more in­tim­id­at­ing at the point when I first ap­proached the prob­lem.

And I real­ized that the word “im­possible” had two us­ages:

1) Mathem­at­ical proof of im­possib­il­ity con­di­tional on spe­cified ax­ioms;

2) “I can’t see any way to do that.”

Need­less to say, all my own uses of the word “im­possible” had been of the second type.

Any time you don’t un­der­stand a do­main, many prob­lems in that do­main will seem im­possible be­cause when you query your brain for a solu­tion path­way, it will re­turn null. But there are only mys­ter­i­ous ques­tions, never mys­ter­i­ous an­swers. If you spend a year or two work­ing on the do­main, then, if you don’t get stuck in any blind al­leys, and if you have the nat­ive abil­ity level re­quired to make pro­gress, you will un­der­stand it bet­ter. The ap­par­ent dif­fi­culty of prob­lems may go way down. It won’t be as scary as it was to your novice-self.

And this is es­pe­cially likely on the con­fus­ing prob­lems that seem most in­tim­id­at­ing.

Since we have some no­tion of the pro­cesses by which a star burns, we know that it’s not easy to build a star from scratch. Be­cause we un­der­stand gears, we can prove that no col­lec­tion of gears obey­ing known phys­ics can form a per­petual mo­tion ma­chine. These are not good prob­lems on which to prac­tice do­ing the im­possible.

When you’re con­fused about a do­main, prob­lems in it will feel very in­tim­id­at­ing and mys­ter­i­ous, and a query to your brain will pro­duce a count of zero solu­tions. But you don’t know how much work will be left when the con­fu­sion clears. Dis­solv­ing the con­fu­sion may it­self be a very dif­fi­cult chal­lenge, of course. But the word “im­possible” should hardly be used in that con­nec­tion. Con­fu­sion ex­ists in the map, not in the ter­rit­ory.

So if you spend a few years work­ing on an im­possible prob­lem, and you man­age to avoid or climb out of blind al­leys, and your nat­ive abil­ity is high enough to make pro­gress, then, by golly, after a few years it may not seem so im­possible after all.

But if some­thing seems im­possible, you won’t try.

Now that’s a vi­cious cycle.

If I hadn’t been in a suf­fi­ciently driven frame of mind that “forty years and a Man­hat­tan Pro­ject” just meant we should get star­ted earlier, I wouldn’t have tried. I wouldn’t have stuck to the prob­lem. And I wouldn’t have got­ten a chance to be­come less in­tim­id­ated.

I’m not or­din­ar­ily a fan of the the­ory that op­pos­ing bi­ases can can­cel each other out, but some­times it hap­pens by luck. If I’d seen that whole moun­tain at the start—if I’d real­ized at the start that the prob­lem was not to build a seed cap­able of im­prov­ing it­self, but to pro­duce a prov­ably cor­rect Friendly AI—then I prob­ably would have burst into flames.

Even so, part of un­der­stand­ing those above-av­er­age sci­ent­ists who con­sti­tute the bulk of AGI re­search­ers, is real­iz­ing that they are not driven to take on a nearly im­possible prob­lem even if it takes them 40 years. By and large, they are there be­cause they have found the Key to AI that will let them solve the prob­lem without such tre­mend­ous dif­fi­culty, in just five years.

Richard Ham­ming used to go around ask­ing his fel­low sci­ent­ists two ques­tions: “What are the im­port­ant prob­lems in your field?“, and, “Why aren’t you work­ing on them?”

Often the im­port­ant prob­lems look Big, Scary, and In­tim­id­at­ing. They don’t prom­ise 10 pub­lic­a­tions a year. They don’t prom­ise any pro­gress at all. You might not get any re­ward after work­ing on them for a year, or five years, or ten years.

And not un­com­monly, the most im­port­ant prob­lems in your field are im­possible. That’s why you don’t see more philo­soph­ers work­ing on re­duc­tion­ist de­com­pos­i­tions of con­scious­ness.

Try­ing to do the im­possible is def­in­itely not for every­one. Ex­cep­tional tal­ent is only the ante to sit down at the table. The chips are the years of your life. If wager­ing those chips and los­ing seems like an un­bear­able pos­sib­il­ity to you, then go do some­thing else. Ser­i­ously. Be­cause you can lose.

I’m not go­ing to say any­thing like, “Every­one should do some­thing im­possible at least once in their life­times, be­cause it teaches an im­port­ant les­son.” Most of the people all of the time, and all of the people most of the time, should stick to the pos­sible.

Never give up? Don’t be ri­dicu­lous. Do­ing the im­possible should be re­served for very spe­cial oc­ca­sions. Learn­ing when to lose hope is an im­port­ant skill in life.

But if there’s some­thing you can ima­gine that’s even worse than wast­ing your life, if there’s some­thing you want that’s more im­port­ant than thirty chips, or if there are scar­ier things than a life of in­con­veni­ence, then you may have cause to at­tempt the im­possible.

There’s a good deal to be said for per­sever­ing through dif­fi­culties; but one of the things that must be said of it, is that it does keep things dif­fi­cult. If you can’t handle that, stay away! There are easier ways to ob­tain glamor and re­spect. I don’t want any­one to read this and need­lessly plunge head­long into a life of per­man­ent dif­fi­culty.

But to con­clude: The “per­sever­ance” that is re­quired to work on im­port­ant prob­lems has a com­pon­ent bey­ond work­ing 14 hours a day.

It’s strange, the pat­tern of what we no­tice and don’t no­tice about ourselves. This se­lectiv­ity isn’t al­ways about in­flat­ing your self-im­age. So­me­times it’s just about or­din­ary sa­li­ence.

To keep work­ing was a con­stant struggle for me, so it was sa­li­ent: I no­ticed that I couldn’t work for 14 solid hours a day. It didn’t oc­cur to me that “per­sever­ance” might also ap­ply at a times­cale of seconds or years. Not un­til I saw people who in­stantly de­clared “im­possible” any­thing they didn’t want to try, or saw how re­luct­ant they were to take on work that looked like it might take a couple of dec­ades in­stead of “five years”.

That was when I real­ized that “per­sever­ance” ap­plied at mul­tiple time scales. On the times­cale of seconds, per­sever­ance is to “not to give up in­stantly at the very first sign of dif­fi­culty”. On the times­cale of years, per­sever­ance is to “keep work­ing on an in­sanely dif­fi­cult prob­lem even though it’s in­con­veni­ent and you could be get­ting higher per­sonal re­wards else­where”.

To do things that are very dif­fi­cult or “im­possible”,

First you have to not run away. That takes seconds.

Then you have to work. That takes hours.

Then you have to stick at it. That takes years.

Of these, I had to learn to do the first re­li­ably in­stead of sporad­ic­ally; the second is still a con­stant struggle for me; and the third comes nat­ur­ally.