Why would we think artists perform better on drugs ?

Introduction

It is com­mon knowl­edge that many artists have used drugs (al­co­hol, opi­ates, cannabis, LSD, …) and that this ac­count for part of their cre­ativity. This com­mon knowl­edge is usu­ally op­posed to peo­ple ad­vo­cat­ing ra­tio­nal­ity in sen­tences like “but with only your ra­tio­nal­ity, we wouldn’t have much art”, “you need chaos to make art” or even “the best artists were that great be­cause they were ir­ra­tional”.

Eliezer partly ad­dressed the is­sue in the lawful in­tel­li­gence Se­quence. While this Se­quence is very in­ter­est­ing, I feel it didn’t com­pletely ad­dress the is­sue (un­like most of the Se­quences). My hy­poth­e­sis is that it’s mostly fo­cused on what’s im­por­tant to build­ing a Friendly AI (which is a wor­thy goal, this should not be taken as a crit­i­cism), not so much as ex­plain­ing cre­ativity in ac­tual hu­mans. So I’m writ­ing an ar­ti­cle with my cur­rent thoughts on the topic, and I would wel­come any ad­di­tional ar­gu­ment, hy­poth­e­sis, re­search pa­per, … that any­one from the LW com­mu­nity can point me to. This ar­ti­cle is not sup­posed to come to any defini­tive con­clu­sion, but to show my cur­rent state of think­ing on that is­sue. I hope to both give and re­ceive in writ­ing it.

Rea­sons for which it could be an illusion

Availa­bil­ity bias

The first ques­tion to ask about “it is a com­mon knowl­edge that many artists were us­ing drugs” is : is this com­mon knowl­edge true, or not, and if not, why do so many peo­ple be­lieve some­thing which is false ?

Availa­bil­ity bias comes will full power on this is­sue : when we hear that a given artist (mu­si­cian, writer, painter, …) was tak­ing drugs, we add a “drug ad­dict” tag to him. Or more ac­cu­rately we cre­ate a link be­tween the “drug ad­dict” node and his node in our be­lief net­work. When asked about artists who did take drugs, we can eas­ily state many names : for ex­am­ple Hem­ing­way, Van Gogh, the Bea­tles. When asked about artists who didn’t take drugs… well, we usu­ally don’t have “did not take drug” node in our be­lief net­works, and no easy way to say that Asi­mov or Bach didn’t take drugs.

Even when do­ing spe­cific re­search, we can know with al­most ab­solute cer­tainty that Hem­ing­way was drink­ing a lot of al­co­hol, but not so con­fi­dently that Asi­mov didn’t. It’s eas­ier to be sure of the ex­is­tence of some­thing, than to be sure of its non-ex­is­tence.

Re­v­erse causality

The sec­ond ques­tion, if even af­ter con­sid­er­ing the af­fect of availa­bil­ity bias, it still seems than artists take drugs more of­ten than av­er­age, is to ask about which sens the causal­ity flows. Statis­ti­cal cor­re­la­tion points to a causal­ity, but doesn’t tell you which sense is the causal­ity, nor if it’s di­rect or in­di­rect.

There can be many rea­sons for which the causal­ity works back­wards : some­one is not a good artist be­cause he takes drug, but he takes drugs be­cause he is an artist.

The lifestyle of a pro­fes­sional artist is usu­ally differ­ent from the lifestyle of most other peo­ple. They usu­ally don’t have to wake up at 7 to be at work at 8, since they can work at any time. They also tend to be ei­ther very poor (many artists were only praised and rec­og­nized af­ter be­ing dead) or very rich (for the few who reach suc­cess while they are still al­ive). And we know that very poor peo­ple tend to fall on al­co­hol more of­ten, while very rich peo­ple tend to use more fre­quently some of the very ex­pen­sive drugs like co­caine.

Be­ing an artist also usu­ally in­duces a higher un­cer­tainty about the fu­ture than with most reg­u­lar jobs, which may trig­ger the use of drugs to make the angst eas­ier to with­stand.

Com­mon cause

Apart from di­rect causal­ity one way or an­other, a statis­ti­cal cor­re­la­tion can also in­di­cate that is a hid­den com­mon cause be­tween the two phe­nom­e­nas. If artists take drugs more of­ten, it could be be­cause there is a com­mon rea­son that pushes peo­ple to both by a great artist and to take drugs.

Many rea­sons can be in­voked that way, due to su­per­ex­po­nen­tial hy­poth­e­sis space. I’ll risk to be priv­ileg­ing the hy­poth­e­sis but I can name a few. For ex­am­ple some­one with an overde­velop emo­tional sen­si­tivity could be both great at writ­ing art able to call to our emo­tions and be more tempted to use drugs as re­lief from over-ex­pe­rienced nega­tive emo­tions. Or some­one who hap­pens to be an out­cast can be more likely to perform art (since it is usu­ally a soli­tary work, not a team work) and at the same time use more drugs to es­cape from the pain of be­ing an out­cast.

So, where do we stand now ?

When faced with a state­ment such as “artists take drugs more of­ten than av­er­age, so drugs help cre­ativity” we can emit 4 differ­ent classes of hy­poth­e­sis :

  1. The ini­tial state­ment is wrong, artists don’t take more drugs than av­er­age.

  2. Artists take more drugs than av­er­age, but the causal­ity is re­verse (it’s be­ing an artist that make you take drugs, not the other way around).

  3. Artists take more drugs than av­er­age, but that’s be­cause of a com­mon fac­tor that in­creases like­li­hood of tak­ing drugs and of mak­ing great art, not the drugs them­selves in­creas­ing artis­tic cre­ativity.

  4. This is true, for a rea­son or an­other, drugs help cre­ativity.

We saw some pos­si­ble rea­sons for 1., 2. and 3. Some of them seem to be very real to me, es­pe­cially the availa­bil­ity bias, but I do not think they to­tally ac­count for the facts.

As much as I would love to be able to stop here and say that drugs and chaos play no pos­i­tive role in cre­ativity, that cre­ativity is purely lawful and ra­tio­nal, I fear that would be wish­ful think­ing and re­fus­ing to at­tack my be­lief’s weak points. To state it more lightly : my D&D al­ign­ment could very well be lawful-good (as my friends tease me it is), but that shouldn’t pre­vent me from ad­mit­ting that chaos play a pos­i­tive role some­where if it ac­tu­ally does.

Rea­sons for which it could be real

Chaos and optimization

Gen­er­at­ing great art can be seen as an op­ti­miza­tion pro­cess. The ac­tual func­tion that eval­u­ate a piece of art may be very com­plex, partly de­pend­ing of the re­cip­i­ent, and its for­mal­iza­tion un­known, but it can still be con­sid­ered an op­ti­miza­tion pro­cess : gen­er­at­ing a book, or a paint­ing, or a song that scores very high in most peo­ple’s eval­u­a­tion func­tion.

In gen­eral, chaos is not an op­ti­miza­tion pro­cess. Ad­ding chaos to an op­ti­miza­tion pro­cess usu­ally makes it worse. But there are known counter-ex­am­ples, where an im­perfect op­ti­miza­tion pro­cess will gain from a slight con­trol­led in­crease of chaos.

Lawfully con­trol­led chaotic optimization

The first known ex­am­ple is the first op­ti­miza­tion pro­cess ever : evolu­tion. Evolu­tion in­volves too part : mu­ta­tions which are chaotic, done at ran­dom, and nat­u­ral se­lec­tion which is lawful and se­lects the few evolu­tions that hap­pened to be pos­i­tive. The Roger Ze­lazny pic­ture of the uni­verse be­ing an equil­ibrium be­tween Order and Chaos may come from that pat­tern. If you in­crease chaos too much in nat­u­ral se­lec­tion, the in­for­ma­tion will not be repli­cated enough from gen­er­a­tion to gen­er­a­tion, and not much op­ti­miza­tion will oc­cur. But if you don’t have any mu­ta­tion, if you re­move all the chaos, the pro­cess will freeze too.

I re­mem­ber an ex­per­i­ment from biol­ogy les­sons in high school : take two small boxes of glass, put cot­ton with wa­ter and sugar at the bot­tom. Take some bac­te­ria, and but the bac­te­ria on side of the box. Take an an­tibiotics pill, and put it on the other side of the box. Put box A in safe stor­age. Put box B in safe stor­age too, but ev­ery day, ex­pose it to a small amount of UV light. The bac­te­ria of box A will quickly spread on the cot­ton, but will not go any­where close the an­tibiotics pill. The bac­te­ria of box B will start do­ing the same, but af­ter two or three weeks, they con­quer even the an­tibiotics area. After a longer time pe­riod, box A bac­te­ria will also over­come the an­tibiotics, but it’ll take them much longer. The UV light in­creased the mu­ta­tion rate, and sped up the op­ti­miza­tion pro­cess of evolu­tion. But only a very small dose of UV light does that, over­dose it, and the bac­te­ria B will all die.

That’s what I call “lawfully con­trol­led chaotic op­ti­miza­tion” : there is a lawful con­trol pro­cess (here, nat­u­ral se­lec­tion) that se­lects ran­domly tried solu­tions. That’s some­thing that can di­rectly ap­ply to artists : the con­trol pro­cess (be it the filter of ed­i­tors/​pub­lish­ers, or the filter of pub­lic re­ac­tion) is, to a point, lawful, but the pro­cess that gen­er­ate solu­tions could benefit from a slight in­crease in chaos. Or more ex­actly, the com­bined (gen­er­a­tor + filter) al­gorithm could perform bet­ter with a slightly more chaotic gen­er­a­tor. To re­take Eliezer’s defi­ni­tion of cre­ativity, which was “the cre­ative sur­prise is the idea that ranks high in your prefer­ence or­der­ing but low in your search or­der­ing”, adding chaos to prefer­ence or­der­ing would be pointless, but adding chaos to the search or­der­ing can al­low more cre­ative sur­prise to hap­pen in a given finite time.

There is still a ma­jor differ­ence be­tween the two pro­cesses de­scribed here (evolu­tion and hu­man cre­ativity) : evolu­tion uses a fully ran­dom gen­er­a­tor, whereas the hu­man brain has a great abil­ity of gen­er­at­ing non-ran­dom de­signs, al­low­ing a much faster im­prove­ment rate. You’ll never get a book of Hem­ing­way or a paint­ing of Van Gogh by ran­domly se­lect­ing let­ters or ran­domly putting paint on a can­vas. The chance of that is too in­finites­i­mally low. So the gen­er­a­tor will have to stay mostly lawful. Hem­ing­way used words and re­spected the rules of gram­mar. Van Gogh painted some­thing which look very like real sun­flow­ers. A fully chaotic pro­cess would never pro­duce any­thing near their mas­ter­pieces even given billions of years. So artis­tic cre­ativity must be mostly lawful, even for gen­er­at­ing its hy­poth­e­sis to se­lect from.

As spot­ted by Vaniver in the com­ments, Hem­ing­way him­self said some­thing very similar to that the­sis : “Write drunk; edit sober.”

Avoid­ing lo­cal minimal

One big prob­lem of op­ti­miza­tion pro­cesses is lo­cal min­i­mal. Most of the naive op­ti­miza­tion pro­cess, like a gra­di­ent de­scent, will get trapped into lo­cal min­ima. Let’s have a look at that curve (bur­rowed from Wikipe­dia) :

Local and global maxima, from Wikipedia

If you start a naive op­ti­miza­tion al­gorithm in the right part of the curve, you’ll very likely end up in the lo­cal min­i­mum, while the global min­i­mum would rank much bet­ter in your prefer­ences. Ad­ding some form of con­trol­led chaos to the al­gorithm is an easy way to in­crease the chance of reach­ing the global min­i­mum, even in much more com­plex se­tups than this sim­ple curve.

For a rel­a­tively broad class of prob­lems, like se­lect­ing the best po­si­tion of nodes to min­i­mize the length of edges when do­ing a bitmap rep­re­sen­ta­tion of a graph struc­ture, an al­gorithm which works quite well and is sim­ple to code is the simu­lated an­neal­ing al­gorithm, which works by do­ing lo­cal op­ti­miza­tions, but hav­ing a global tem­per­a­ture which adds chaos (the higher the tem­per­a­ture, the more ran­dom is the pro­cess). The tem­per­a­ture it­self de­creases with the pro­cess, and ul­ti­mately reaches 0 (pure lawful op­ti­miza­tion).

Such meth­ods are of course “dirty hacks”, that are used only when the prob­lem is too com­pli­cated and we don’t have a purely lawful al­gorithm that gives the an­swer, or (most of the time) when we do have one, but with an ex­po­nen­tial com­plex­ity, mean­ing we can’t run it in real life.

The same idea ap­plies to hu­man cre­ativity : chaos wouldn’t be needed, nor use­ful, if we had a fully work­ing al­gorithm to write the best books or songs or make the best paint­ings. But since we don’t, us­ing a purely lawful pro­cess has a risk (but yes, only a risk) of get­ting us stuck into a lo­cal min­i­mum—im­prov­ing the meth­ods of the pre­vi­ous gen­er­a­tion of artists, but not in­vent­ing brand new styles of art. This is a similar con­cept to the “jump­ing out of the sys­tem” de­scribed by Dou­glas Hofs­tadter and an­a­lyzed by Eliezer. JOOTSing is es­cap­ing a lo­cal min­i­mum. It’s es­cap­ing the safe warmth of the valley, climb­ing the cold and dan­ger­ous moun­tain top, to find an­other, more fer­tile valley on the other side. That re­quires to vi­o­late the rules of “stay­ing into the safe and warm valley”.

(Note : there is some­how an anal­ogy be­tween the use of drugs and the simu­lated an­neal­ing : drugs in­duce a state of high chaos, which then slowly goes down as the drug effects dis­ap­pears. Or at least I was told so, since I never tried per­son­ally. But that seems a sur­face anal­ogy to me, so I won’t give it much credit).

In­hi­bi­tions and art

Or an­other way to con­sider it is to look at is in­hi­bi­tions : the hu­man mind con­tains a pro­cess that’ll check your ac­tions (paint­ing and writ­ing in that case, but ap­plies more broadly) and some­times say “no, don’t do that, you’ll look as a fool”. Those in­hi­bi­tions are usu­ally here to pro­tect us from botch­ing in so­cial situ­a­tions. But they are (as most of the hu­man brain) im­perfectly cal­ibrated, and will tend to re­press any­thing that goes out of the cur­rent norm. Low­er­ing those in­hi­bi­tions in­creases the risk of botch­ing—but also the chance of do­ing some­thing awe­some.

This points to much more gen­eral pat­tern, which ap­plies when what mat­ters is not im­prov­ing your av­er­age gain, but your chance of be­ing one of the few best. Con­sider you’ve a task to do, and two ways to achieve it. Way A is quite clas­si­cal, and doesn’t in­volve much risk. Way B is much less proven, and in­volves much risk of do­ing both bet­ter and worse. Be­ing a role player, I usu­ally use dice rolls to model those kinds of pro­cess. Let’s say pro­cess A is 20d10. That means, rol­ling 20 times a 10-sided die, and do­ing the sum. This will give an ex­pected value of 110, with only 1% of the rolls above 140. Now pro­cess B is 2d100 (rol­ling 2 times a 100-sided die and do­ing the sum). This will give a lower ex­pected value, of 101 in­stead of 110. But with 18% of the rolls above 140. Here is a pic­ture of the two pro­cess (way A in green, way B in red) gen­er­ated with a quick Python script :

2d100 (red) vs 20d10 (green)

If what mat­ters is do­ing your best in av­er­age (your score at the task will di­rectly map to an amount of money be­tween $2 and $200), then the best choice is to look only at the ex­pected value of A and B, and se­lect the one which has the best ex­pected value, so A in this case, as you can see, the green curve peaks at a higher value.

But if what mat­ters is not do­ing the best in av­er­age, but be­ing the best : 100 peo­ple are perform­ing the task, and the best will take the prize, the rest won’t have any­thing. Then, you ex­cept one of the 100 to be above 140, even if they all use way A. So for your­self, if you use way A, you only have 1 chance in 100 to be above 140. If you use way B, you’ve 18 chances in 100 of beat­ing the 140 mark. Look­ing at the curve, there is much big­ger blob of the red curve that goes to very high val­ues.

When look­ing at arts, we don’t re­gard the av­er­age. Countless peo­ple write books or paint. Al­most ev­ery­one at least tried once. What his­tory re­mem­bers are the few best of their time. Not those who did bet­ter in av­er­age, but those who man­age to do bet­ter than most of their peers. Those to the right of the pic­ture, in which the am­pli­tude of the green curve is nearly void, but the red curve still ex­ists.

The com­plex­ity of test­ing cer­tain hypothesis

I emit­ted many hy­poth­e­sis in this ar­ti­cle, to try to ex­plain the com­mon knowl­edge that “so many great artists take drugs”, and more gen­er­ally to look into the rea­sons for which chaos can, in some cases, im­prove a re­sult.

All those hy­poth­e­sis seem to­tally plau­si­ble to me—and I would say that they all play a role in the pro­cess. But say­ing “ev­ery­thing plays a role” is not say­ing much, a graph with all pos­si­ble edges con­tains as much in­for­ma­tions as a graph with no edge. What would be re­quire now is to con­sider how much each hy­poth­e­sis con­tributes to the re­sult—and then, prob­a­bly one or two will ac­count for most.

But how can we setup such a test ? In physics, do­ing ex­per­i­ments is rel­a­tively easy. It can costs a lot like build­ing the LHC or send­ing the Hub­ble space telescope in or­bit, but still, de­vis­ing ex­per­i­ments is rel­a­tively easy. In so­cial sci­ences, it’s of­ten much harder. Most so­cial sci­ence ex­per­i­ments are done on a panel of test sub­jects (with a con­trol group, …). But right now we are speak­ing of the best artists. How can we build such a panel ? Defin­ing who are the best artists is a very hard task. And then, get­ting them to par­ti­ci­pate in stud­ies...

The sim­plest hy­poth­e­sis to test, the availa­bil­ity bias, would re­quire a pro­ce­dure like (num­bers can be ad­justed) :

  1. Take 1000 peo­ple at ran­dom, from var­i­ous ages, so­cial classes and back­ground.

  2. Ask for each of them to name the 10 artists they like the most (with­out of course men­tion­ing the pur­pose of the ex­per­i­ment).

  3. For each artist nom­i­nated by at least 4 per­sons, look if that artist did take drugs.

  4. Com­pare with the av­er­age drug use.

But even that is not with­out trou­bles : for 3., how can you be sure an artist didn’t take drug se­cretly, es­pe­cially in time/​places where drug use is pro­hibited or frown upon ? For 4., how do you pon­der for the vari­a­tion in drug use de­pend­ing of the place/​time ?

Does any­one know of such a study (I couldn’t find any, but I’m not well versed in the art for look­ing for so­cial sci­ence stud­ies) ?

For the other hy­poth­e­sis, test­ing them be­comes even harder.

Conclusion

As Eliezer ex­plained, pure chaos can­not lead to any­thing but static on a TV screen. Any op­ti­miza­tion pro­cess, and art is one, re­quires a lawful part. But as I showed, for sev­eral rea­sons, an im­perfect op­ti­miza­tion pro­cess may perform bet­ter with a limited amount of added chaos. Since the hu­man brain is an im­perfect op­ti­miza­tion pro­cess, it would not be sur­pris­ing that in the pur­pose of cre­at­ing awe­some pieces of art in a limited time, some added chaos can help. But on the other hand, there are other rea­sons for which there could be a com­mon knowl­edge that “artis­tic cre­ativity re­quires some chaos” even if it were not true. And it is very hard to tell apart the var­i­ous rea­sons.

But even if some amount of chaos can help in gen­er­at­ing ex­cep­tion­ally awe­some pieces of art, it should not shadow the fact that the lawful part of pro­cess is ab­solutely re­quired, and even the most im­por­tant one; nor that chaos can only be use­ful when the op­ti­miza­tion is it­self im­perfect. Im­prov­ing the qual­ity of the op­ti­miza­tion pro­cess (by, for ex­am­ple, rais­ing the san­ity wa­ter­line or un­der­stand­ing bet­ter the hu­man brain) would lower the need of chaos to gen­er­ate the same awe­some­ness.

PS : I post that to “Less Wrong dis­cus­sion”, for ini­tial re­view and be­cause it’s half-way be­tween a “real” ar­ti­cle and a call for dis­cus­sion on the topic. Depend­ing of feed­back, I hope to re­post it to “main Less Wrong”, hope­fully im­proved from the feed­back.