On Doing the Impossible

“Per­se­vere.” It’s a piece of ad­vice you’ll get from a whole lot of high achiev­ers in a whole lot of dis­ci­plines. I didn’t un­der­stand it at all, at first.

At first, I thought “per­se­ver­ance” meant work­ing 14-hour days. Ap­par­ently, there are peo­ple out there who can work for 10 hours at a tech­ni­cal job, and then, in their mo­ments be­tween eat­ing and sleep­ing and go­ing to the bath­room, seize that un­filled spare time to work on a book. I am not one of those peo­ple—it still hurts my pride even now to con­fess that. I’m work­ing on some­thing im­por­tant; shouldn’t my brain be will­ing to put in 14 hours a day? But it’s not. When it gets too hard to keep work­ing, I stop and go read or watch some­thing. Be­cause of that, I thought for years that I en­tirely lacked the virtue of “per­se­ver­ance”.

In ac­cor­dance with hu­man na­ture, Eliezer1998 would think things like: “What counts is out­put, not in­put.” Or, “Laz­i­ness is also a virtue—it leads us to back off from failing meth­ods and think of bet­ter ways.” Or, “I’m do­ing bet­ter than other peo­ple who are work­ing more hours. Maybe, for cre­ative work, your mo­men­tary peak out­put is more im­por­tant than work­ing 16 hours a day.” Per­haps the fa­mous sci­en­tists were se­duced by the Deep Wis­dom of say­ing that “hard work is a virtue”, be­cause it would be too awful if that counted for less than in­tel­li­gence?

I didn’t un­der­stand the virtue of per­se­ver­ance un­til I looked back on my jour­ney through AI, and re­al­ized that I had over­es­ti­mated the difficulty of al­most ev­ery sin­gle im­por­tant prob­lem.

Sounds crazy, right? But bear with me here.

When I was first de­cid­ing to challenge AI, I thought in terms of 40-year timescales, Man­hat­tan Pro­jects, plane­tary com­put­ing net­works, mil­lions of pro­gram­mers, and pos­si­bly aug­mented hu­mans.

This is a com­mon failure mode in AI-fu­tur­ism which I may write about later; it con­sists of the leap from “I don’t know how to solve this” to “I’ll imag­ine throw­ing some­thing re­ally big at it”. Some­thing huge enough that, when you imag­ine it, that imag­i­na­tion cre­ates a feel­ing of im­pres­sive­ness strong enough to be com­men­su­rable with the prob­lem. (There’s a fel­low cur­rently on the AI list who goes around say­ing that AI will cost a quadrillion dol­lars—we can’t get AI with­out spend­ing a quadrillion dol­lars, but we could get AI at any time by spend­ing a quadrillion dol­lars.) This, in turn, lets you imag­ine that you know how to solve AI, with­out try­ing to fill the ob­vi­ously-im­pos­si­ble de­mand that you un­der­stand in­tel­li­gence.

So, in the be­gin­ning, I made the same mis­take: I didn’t un­der­stand in­tel­li­gence, so I imag­ined throw­ing a Man­hat­tan Pro­ject at the prob­lem.

But, hav­ing calcu­lated the plane­tary death rate at 55 mil­lion per year or 150,000 per day, I did not turn around and run away from the big scary prob­lem like a fright­ened rab­bit. In­stead, I started try­ing to figure out what kind of AI pro­ject could get there fastest. If I could make the Sin­gu­lar­ity hap­pen one hour ear­lier, that was a rea­son­able re­turn on in­vest­ment for a pre-Sin­gu­lar­ity ca­reer. (I wasn’t think­ing in terms of ex­is­ten­tial risks or Friendly AI at this point.)

So I didn’t run away from the big scary prob­lem like a fright­ened rab­bit, but stayed to see if there was any­thing I could do.

Fun his­tor­i­cal fact: In 1998, I’d writ­ten this long trea­tise propos­ing how to go about cre­at­ing a self-im­prov­ing or “seed” AI (a term I had the honor of coin­ing). Brian Atk­ins, who would later be­come the found­ing fun­der of the Sin­gu­lar­ity In­sti­tute, had just sold Hyper­mart to Go2Net. Brian emailed me to ask whether this AI pro­ject I was de­scribing was some­thing that a rea­son­able-sized team could go out and ac­tu­ally do. “No,” I said, “it would take a Man­hat­tan Pro­ject and thirty years,” so for a while we were con­sid­er­ing a new dot-com startup in­stead, to cre­ate the fund­ing to get real work done on AI...

A year or two later, af­ter I’d heard about this new­fan­gled “open source” thing, it seemed to me that there was some pre­limi­nary de­vel­op­ment work—new com­puter lan­guages and so on—that a small or­ga­ni­za­tion could do; and that was how the Sin­gu­lar­ity In­sti­tute started.

This strat­egy was, of course, en­tirely wrong.

But even so, I went from “There’s noth­ing I can do about it now” to “Hm… maybe there’s an in­cre­men­tal path through open-source de­vel­op­ment, if the ini­tial ver­sions are use­ful to enough peo­ple.”

This is back at the dawn of time, so I’m not say­ing any of this was a good idea. But in terms of what I thought I was try­ing to do, a year of cre­ative think­ing had short­ened the ap­par­ent path­way: The prob­lem looked slightly less im­pos­si­ble than it did the very first time I ap­proached it.

The more in­ter­est­ing pat­tern is my en­try into Friendly AI. Ini­tially, Friendly AI hadn’t been some­thing that I had con­sid­ered at all—be­cause it was ob­vi­ously im­pos­si­ble and use­less to de­ceive a su­per­in­tel­li­gence about what was the right course of ac­tion.

So, his­tor­i­cally, I went from com­pletely ig­nor­ing a prob­lem that was “im­pos­si­ble”, to tak­ing on a prob­lem that was merely ex­tremely difficult.

Nat­u­rally this in­creased my to­tal work­load.

Same thing with try­ing to un­der­stand in­tel­li­gence on a pre­cise level. Origi­nally, I’d writ­ten off this prob­lem as im­pos­si­ble, thus re­mov­ing it from my work­load. (This logic seems pretty de­ranged in ret­ro­spect—Na­ture doesn’t care what you can’t do when It’s writ­ing your pro­ject re­quire­ments—but I still see AIfolk try­ing it all the time.) To hold my­self to a pre­cise stan­dard meant putting in more work than I’d pre­vi­ously imag­ined I needed. But it also meant tack­ling a prob­lem that I would have dis­missed as en­tirely im­pos­si­ble not too much ear­lier.

Even though in­di­vi­d­ual prob­lems in AI have seemed to be­come less in­timi­dat­ing over time, the to­tal moun­tain-to-be-climbed has in­creased in height—just like con­ven­tional wis­dom says is sup­posed to hap­pen—as prob­lems got taken off the “im­pos­si­ble” list and put on the “to do” list.

I started to un­der­stand what was hap­pen­ing—and what “Per­se­vere!” re­ally meant—at the point where I no­ticed other AIfolk do­ing the same thing: say­ing “Im­pos­si­ble!” on prob­lems that seemed em­i­nently solv­able—rel­a­tively more straight­for­ward, as such things go. But they were things that would have seemed vastly more in­timi­dat­ing at the point when I first ap­proached the prob­lem.

And I re­al­ized that the word “im­pos­si­ble” had two us­ages:

1) Math­e­mat­i­cal proof of im­pos­si­bil­ity con­di­tional on speci­fied ax­ioms;

2) “I can’t see any way to do that.”

Need­less to say, all my own uses of the word “im­pos­si­ble” had been of the sec­ond type.

Any time you don’t un­der­stand a do­main, many prob­lems in that do­main will seem im­pos­si­ble be­cause when you query your brain for a solu­tion path­way, it will re­turn null. But there are only mys­te­ri­ous ques­tions, never mys­te­ri­ous an­swers. If you spend a year or two work­ing on the do­main, then, if you don’t get stuck in any blind alleys, and if you have the na­tive abil­ity level re­quired to make progress, you will un­der­stand it bet­ter. The ap­par­ent difficulty of prob­lems may go way down. It won’t be as scary as it was to your novice-self.

And this is es­pe­cially likely on the con­fus­ing prob­lems that seem most in­timi­dat­ing.

Since we have some no­tion of the pro­cesses by which a star burns, we know that it’s not easy to build a star from scratch. Be­cause we un­der­stand gears, we can prove that no col­lec­tion of gears obey­ing known physics can form a per­pet­ual mo­tion ma­chine. Th­ese are not good prob­lems on which to prac­tice do­ing the im­pos­si­ble.

When you’re con­fused about a do­main, prob­lems in it will feel very in­timi­dat­ing and mys­te­ri­ous, and a query to your brain will pro­duce a count of zero solu­tions. But you don’t know how much work will be left when the con­fu­sion clears. Dis­solv­ing the con­fu­sion may it­self be a very difficult challenge, of course. But the word “im­pos­si­ble” should hardly be used in that con­nec­tion. Con­fu­sion ex­ists in the map, not in the ter­ri­tory.

So if you spend a few years work­ing on an im­pos­si­ble prob­lem, and you man­age to avoid or climb out of blind alleys, and your na­tive abil­ity is high enough to make progress, then, by golly, af­ter a few years it may not seem so im­pos­si­ble af­ter all.

But if some­thing seems im­pos­si­ble, you won’t try.

Now that’s a vi­cious cy­cle.

If I hadn’t been in a suffi­ciently driven frame of mind that “forty years and a Man­hat­tan Pro­ject” just meant we should get started ear­lier, I wouldn’t have tried. I wouldn’t have stuck to the prob­lem. And I wouldn’t have got­ten a chance to be­come less in­timi­dated.

I’m not or­di­nar­ily a fan of the the­ory that op­pos­ing bi­ases can can­cel each other out, but some­times it hap­pens by luck. If I’d seen that whole moun­tain at the start—if I’d re­al­ized at the start that the prob­lem was not to build a seed ca­pa­ble of im­prov­ing it­self, but to pro­duce a prov­ably cor­rect Friendly AI—then I prob­a­bly would have burst into flames.

Even so, part of un­der­stand­ing those above-av­er­age sci­en­tists who con­sti­tute the bulk of AGI re­searchers, is re­al­iz­ing that they are not driven to take on a nearly im­pos­si­ble prob­lem even if it takes them 40 years. By and large, they are there be­cause they have found the Key to AI that will let them solve the prob­lem with­out such tremen­dous difficulty, in just five years.

Richard Ham­ming used to go around ask­ing his fel­low sci­en­tists two ques­tions: “What are the im­por­tant prob­lems in your field?“, and, “Why aren’t you work­ing on them?”

Often the im­por­tant prob­lems look Big, Scary, and In­timi­dat­ing. They don’t promise 10 pub­li­ca­tions a year. They don’t promise any progress at all. You might not get any re­ward af­ter work­ing on them for a year, or five years, or ten years.

And not un­com­monly, the most im­por­tant prob­lems in your field are im­pos­si­ble. That’s why you don’t see more philoso­phers work­ing on re­duc­tion­ist de­com­po­si­tions of con­scious­ness.

Try­ing to do the im­pos­si­ble is definitely not for ev­ery­one. Ex­cep­tional tal­ent is only the ante to sit down at the table. The chips are the years of your life. If wa­ger­ing those chips and los­ing seems like an un­bear­able pos­si­bil­ity to you, then go do some­thing else. Se­ri­ously. Be­cause you can lose.

I’m not go­ing to say any­thing like, “Every­one should do some­thing im­pos­si­ble at least once in their life­times, be­cause it teaches an im­por­tant les­son.” Most of the peo­ple all of the time, and all of the peo­ple most of the time, should stick to the pos­si­ble.

Never give up? Don’t be ridicu­lous. Do­ing the im­pos­si­ble should be re­served for very spe­cial oc­ca­sions. Learn­ing when to lose hope is an im­por­tant skill in life.

But if there’s some­thing you can imag­ine that’s even worse than wast­ing your life, if there’s some­thing you want that’s more im­por­tant than thirty chips, or if there are scarier things than a life of in­con­ve­nience, then you may have cause to at­tempt the im­pos­si­ble.

There’s a good deal to be said for per­se­ver­ing through difficul­ties; but one of the things that must be said of it, is that it does keep things difficult. If you can’t han­dle that, stay away! There are eas­ier ways to ob­tain glamor and re­spect. I don’t want any­one to read this and need­lessly plunge head­long into a life of per­ma­nent difficulty.

But to con­clude: The “per­se­ver­ance” that is re­quired to work on im­por­tant prob­lems has a com­po­nent be­yond work­ing 14 hours a day.

It’s strange, the pat­tern of what we no­tice and don’t no­tice about our­selves. This se­lec­tivity isn’t always about in­flat­ing your self-image. Some­times it’s just about or­di­nary salience.

To keep work­ing was a con­stant strug­gle for me, so it was salient: I no­ticed that I couldn’t work for 14 solid hours a day. It didn’t oc­cur to me that “per­se­ver­ance” might also ap­ply at a timescale of sec­onds or years. Not un­til I saw peo­ple who in­stantly de­clared “im­pos­si­ble” any­thing they didn’t want to try, or saw how re­luc­tant they were to take on work that looked like it might take a cou­ple of decades in­stead of “five years”.

That was when I re­al­ized that “per­se­ver­ance” ap­plied at mul­ti­ple time scales. On the timescale of sec­onds, per­se­ver­ance is to “not to give up in­stantly at the very first sign of difficulty”. On the timescale of years, per­se­ver­ance is to “keep work­ing on an in­sanely difficult prob­lem even though it’s in­con­ve­nient and you could be get­ting higher per­sonal re­wards el­se­where”.

To do things that are very difficult or “im­pos­si­ble”,

First you have to not run away. That takes sec­onds.

Then you have to work. That takes hours.

Then you have to stick at it. That takes years.

Of these, I had to learn to do the first re­li­ably in­stead of spo­rad­i­cally; the sec­ond is still a con­stant strug­gle for me; and the third comes nat­u­rally.