Subagents, akrasia, and coherence in humans

In my pre­vi­ous posts, I have been build­ing up a model of mind as a col­lec­tion of sub­agents with differ­ent goals, and no straight­for­ward hi­er­ar­chy. This then raises the ques­tion of how that col­lec­tion of sub­agents can ex­hibit co­her­ent be­hav­ior: af­ter all, many ways of ag­gre­gat­ing the prefer­ences of a num­ber of agents fail to cre­ate con­sis­tent prefer­ence or­der­ings.

We can roughly de­scribe co­her­ence as the prop­erty that, if you be­come aware that there ex­ists a more op­ti­mal strat­egy for achiev­ing your goals than the one that you are cur­rently ex­e­cut­ing, then you will switch to that bet­ter strat­egy. If an agent is not co­her­ent in this way, then bad things are likely to hap­pen to them.

Now, we all know that hu­mans some­times ex­press in­co­her­ent be­hav­ior. But on the whole, peo­ple still do okay: the me­dian per­son in a de­vel­oped coun­try still man­ages to sur­vive un­til their body starts giv­ing up on them, and typ­i­cally also man­ages to have and raise some num­ber of ini­tially-hel­pless chil­dren un­til they are old enough to take care of them­selves.

For a sub­agent the­ory of mind, we would like to have some ex­pla­na­tion of when ex­actly the sub­agents man­age to be col­lec­tively co­her­ent (that is, change their be­hav­ior to some bet­ter one), and what are the situ­a­tions in which they fail to do so. The con­clu­sion of this post will be:

We are ca­pa­ble of chang­ing our be­hav­iors on oc­ca­sions when the mind-sys­tem as a whole puts suffi­ciently high prob­a­bil­ity on the new be­hav­ior be­ing bet­ter, when the new be­hav­ior is not be­ing blocked by a par­tic­u­lar highly weighted sub­agent (such as an IFS-style pro­tec­tor) that puts high prob­a­bil­ity on it be­ing bad, and when we have enough slack in our lives for any new be­hav­iors to be eval­u­ated in the first place. Akra­sia is sub­agent dis­agree­ment about what to do.

(Those of you who read my pre­vi­ous post might re­mem­ber that I said this post would be about “unifi­ca­tion of mind”—that is, about how to make sub­agents agree with each other bet­ter. Turns out that I spent so many words ex­plain­ing when sub­agents dis­agree, that I had to put off the post on how to get them to agree. Maybe my next post will man­age to be about that…)

Cor­rect­ing your be­hav­ior as a default

There are many situ­a­tions in which we ex­hibit in­co­her­ent be­hav­ior sim­ply be­cause we’re not aware of it. For in­stance, sup­pose that I do my daily chores in a par­tic­u­lar or­der, when do­ing them in some other or­der would save more time. If you point this out to me, I’m likely to just say “oh”, and then adopt the bet­ter sys­tem.

Similarly, sev­eral of the ex­per­i­ments which get peo­ple to ex­hibit in­co­her­ent be­hav­ior rely on show­ing differ­ent groups of peo­ple differ­ent for­mu­la­tions of the same ques­tion, and then in­di­cat­ing that differ­ent fram­ings of the same ques­tion get differ­ent an­swers from peo­ple. It doesn’t work quite as well if you show the differ­ent for­mu­la­tions to the same peo­ple, be­cause then many of them will re­al­ize that differ­ing an­swers would be in­con­sis­tent.

But there are also situ­a­tions in which some­one re­al­izes that they are be­hav­ing in a non­sen­si­cal way, yet will con­tinue be­hav­ing in that way. Since peo­ple usu­ally can change sub­op­ti­mal be­hav­iors, we need an ex­pla­na­tion for why they some­times can’t.

Tow­ers of pro­tec­tors as a method for coherence

In my post about In­ter­nal Fam­ily Sys­tems, I dis­cussed a model of mind com­posed of sev­eral differ­ent kinds of sub­agents. One of them, the de­fault plan­ning sub­agent, is a mod­ule just try­ing to straight­for­wardly find the best thing to do and then ex­e­cute that. On the other hand, pro­tec­tor sub­agents ex­ist to pre­vent the sys­tem from get­ting into situ­a­tions which were catas­trophic be­fore. If they think that the de­fault plan­ning sub­agent is do­ing some­thing which seems dan­ger­ous, they will over­ride it and do some­thing else in­stead. (Pre­vi­ous ver­sions of the IFS post called the de­fault plan­ning agent, “a re­in­force­ment learn­ing sub­agent”, but this was po­ten­tially mis­lead­ing since sev­eral other sub­agents were re­in­force­ment learn­ing ones too, so I’ve changed the name.)

Thus, your be­hav­ior can still be co­her­ent even if you feel that you are failing to act in a co­her­ent way. You sim­ply don’t re­al­ize that a pro­tec­tor is car­ry­ing out a rou­tine in­tended to avoid dan­ger­ous out­comes—and this might ac­tu­ally be a very suc­cess­ful way of keep­ing you out of dan­ger. Some sub­agents in your mind think that do­ing X would be a su­pe­rior strat­egy, but the pro­tec­tor thinks that it would be a hor­rible idea—so from the point of view of the sys­tem as a whole, do­ing X is not a bet­ter strat­egy, so not switch­ing to it is ac­tu­ally bet­ter.

On the other hand, it may also be the case that the pro­tec­tor’s be­hav­ior, while keep­ing you out of situ­a­tions which the pro­tec­tor con­sid­ers un­ac­cept­able, is caus­ing other out­comes which are also un­ac­cept­able. The de­fault plan­ning sub­agent may re­al­ize this—but as already es­tab­lished, any pro­tec­tor can over­rule it, so this doesn’t help.

Evolu­tion’s an­swer here seems to be spaghetti tow­ers. The de­fault plan­ning sub­agent might even­tu­ally figure out the bet­ter strat­egy, which avoids both the thing that the pro­tec­tor is try­ing to block and the new bad out­come. But it could be dan­ger­ous to wait that long, es­pe­cially since the de­fault plan­ning agent doesn’t have di­rect ac­cess to the pro­tec­tor’s goals. So for the same rea­sons why a sep­a­rate pro­tec­tor sub­agent was cre­ated to avoid the first catas­tro­phe, the mind will cre­ate or re­cruit a pro­tec­tor to avoid the sec­ond catas­tro­phe—the one that the first pro­tec­tor keeps caus­ing.

With per­mis­sion, I’ll bor­row the illus­tra­tions from eu­kary­ote’s spaghetti tower post to illus­trate this.

Ex­am­ple Eric grows up in an en­vi­ron­ment where he learns that dis­agree­ing with other peo­ple is un­safe, and that he should always agree to do things that other peo­ple ask of him. So Eric de­vel­ops a pro­tec­tor sub­agent run­ning a pleas­ing, sub­mis­sive be­hav­ior.

Un­for­tu­nately, while this tac­tic worked in Eric’s child­hood home, once he be­came an adult he starts say­ing “yes” to too many things, with­out leav­ing any time for his own needs. But say­ing “no” to any­thing still feels un­safe, so he can’t just stop say­ing “yes”. In­stead he de­vel­ops a pro­tec­tor which tries to keep him out of situ­a­tions where peo­ple would ask him to do any­thing. This way, he doesn’t need to say “no”, and also won’t get over­whelmed by all the things that he has promised to do. The two pro­tec­tors to­gether form a com­pos­ite strat­egy.

While this helps, it still doesn’t en­tirely solve the is­sue. After all, there are plenty of rea­sons that might push Eric into situ­a­tions where some­one would ask some­thing of him. He still ends up agree­ing to do lots of things, to the point of ne­glect­ing his own needs. Even­tu­ally, his brain cre­ates an­other pro­tec­tor sub­agent. This one causes ex­haus­tion and de­pres­sion, so that he now has a so­cially-ac­cept­able rea­son for be­ing un­able to do all the things that he has promised to do. He con­tinues say­ing “yes” to things, but also keeps apol­o­giz­ing for be­ing un­able to do things that he (hon­estly) in­tended to do as promised, and even­tu­ally peo­ple re­al­ize that you prob­a­bly shouldn’t ask him to do any­thing that’s re­ally im­por­tant to get done.

And while this kind of a pro­cess of stack­ing pro­tec­tor on top of a pro­tec­tor is not perfect, for most peo­ple it mostly works out okay. Al­most ev­ery­one ends up hav­ing their unique set of minor neu­roses and situ­a­tions where they don’t quite be­have ra­tio­nally, but as they learn to un­der­stand them­selves bet­ter, their de­fault plan­ning sub­agent gets bet­ter at work­ing around those is­sues. This might also make the var­i­ous pro­tec­tors re­lax a bit, since the var­i­ous threats are gen­er­ally avoided and there isn’t a need to keep avoid­ing them.

Grad­u­ally, as nega­tive con­se­quences to differ­ent be­hav­iors be­come ap­par­ent, be­hav­ior gets ad­justed—ei­ther by the de­fault plan­ning sub­agents or by spawn­ing more pro­tec­tors—and re­mains co­her­ent over­all.

But some­times, es­pe­cially for peo­ple in highly stress­ful en­vi­ron­ments where al­most any mis­take may get them pun­ished, or when they end up in an en­vi­ron­ment that their old tower of pro­tec­tors is no longer well-suited for (dis­tri­bu­tional shift), things don’t go as well. In that situ­a­tion, their minds may end up look­ing like this a hope­lessly tan­gled web, where they have al­most no flex­i­bil­ity. Some­thing hap­pens in their en­vi­ron­ment, which sets off one pro­tec­tor, which sets off an­other, which sets off an­other—leav­ing them with no room for flex­i­bil­ity or ra­tio­nal plan­ning, but rather forc­ing them to act in a way which is al­most bound to only make mat­ters worse.

This kind of an out­come is ob­vi­ously bad. So be­sides build­ing spaghetti tow­ers, the sec­ond strat­egy which the mind has evolved to em­ploy for keep­ing its be­hav­ior co­her­ent while piling up pro­tec­tors, is the abil­ity to re-pro­cess mem­o­ries of past painful events.

As I dis­cussed in my origi­nal IFS post, the mind has meth­ods for bring­ing up the origi­nal mem­o­ries which caused a pro­tec­tor to emerge, in or­der to re-an­a­lyze them. If end­ing up in some situ­a­tion is ac­tu­ally no longer catas­trophic (for in­stance, you are no longer in your child­hood home where you get pun­ished sim­ply for not want­ing to do some­thing), then the pro­tec­tors which were fo­cused on avoid­ing that out­come can re­lax and take a less ex­treme role.

For this pur­pose, there seems to be a built-in ten­sion. Ex­iles (the IFS term for sub­agents con­tain­ing mem­o­ries of past trauma) “want” to be healed and will do things like oc­ca­sion­ally send­ing painful mem­o­ries or feel­ings into con­scious­ness so as to be­come the cen­ter of at­ten­tion, es­pe­cially if there is some­thing about the cur­rent situ­a­tion which re­sem­bles the past trauma. This also acts as what my IFS post called a fear model—some­thing that warns of situ­a­tions which re­sem­ble the past trauma enough to be con­sid­ered dan­ger­ous in their own right. At the same time, pro­tec­tors “want” to keep the ex­iles hid­den and in­ac­tive, do­ing any­thing that they can for keep­ing them so. Var­i­ous schools of ther­apy—IFS one of them—seek to tap into this ex­ist­ing ten­sion so as to re­veal the trauma, trace it back to its origi­nal source, and heal it.

Co­her­ence and con­di­tioned responses

Be­sides the pres­ence of pro­tec­tors, an­other pos­si­bil­ity for why we might fail to change our be­hav­ior are strongly con­di­tioned habits. Most hu­man be­hav­ior in­volves au­to­matic habits: be­hav­ioral rou­tines which are trig­gered by some sort of a cue in the en­vi­ron­ment, and lead to or have once led to a re­ward. (Pre­vi­ous dis­cus­sion; see also.)

The prob­lem with this is that peo­ple might end up with habits that they wouldn’t want to have. For in­stance, I might de­velop a habit of check­ing so­cial me­dia on their phone when I’m bored, cre­at­ing a loop of bore­dom (cue) → look­ing at so­cial me­dia (be­hav­ior) → see­ing some­thing in­ter­est­ing on so­cial me­dia (re­ward).

Reflect­ing on this be­hav­ior, I no­tice that back when I didn’t do it, my mind was more free to wan­der when I was bored, gen­er­at­ing mo­ti­va­tion and ideas. I think that my old be­hav­ior was more valuable than my new one. But even so, my new be­hav­ior still de­liv­ers enough mo­men­tary satis­fac­tion to keep re­in­forc­ing the habit.

Sub­jec­tively, this feels like an in­creas­ing com­pul­sion to check my phone, which I try to re­sist since I know that long-term it would be a bet­ter idea to not be check­ing my phone all the time. But as the com­pul­sion keeps grow­ing stronger and stronger, even­tu­ally I give up and look at the phone any­way.

The ex­act neu­ro­science of what is hap­pen­ing at such a mo­ment re­mains only par­tially un­der­stood (Simp­son & Balsam 2016). How­ever, we know that when­ever differ­ent sub­sys­tems in the brain pro­duce con­flict­ing mo­tor com­mands, that con­flict needs to be re­solved, with only one at a time be­ing granted ac­cess to the “fi­nal com­mon mo­tor path”. This is thought to hap­pen in the basal gan­glia, a part of the brain closely in­volved in ac­tion se­lec­tion and con­nected to the global neu­ronal workspace.

One model (e.g. Red­grave 2007, McHaf­fie 2005) is that the basal gan­glia re­ceives in­puts from many differ­ent brain sys­tems; each of those sys­tems can send differ­ent “bids” sup­port­ing or op­pos­ing a spe­cific course of ac­tion to the basal gan­glia. A bid sub­mit­ted by one sub­sys­tem may, through looped con­nec­tions go­ing back from the basal gan­glia, in­hibit other sub­sys­tems, un­til one of the pro­posed ac­tions be­comes suffi­ciently dom­i­nant to be taken.

The above image from Red­grave 2007 has a con­cep­tual image of the model, with two ex­am­ple sub­sys­tems shown. Sup­pose that you are eat­ing at a restau­rant in Juras­sic Park when two ve­lo­cirap­tors charge in through the win­dow. Pre­vi­ously, your hunger sys­tem was sub­mit­ting suc­cess­ful bids for the “let’s keep eat­ing” ac­tion, which then caused in­hibitory im­pulses to the be sent to the threat sys­tem. This in­hi­bi­tion pre­vented the threat sys­tem from mak­ing bids for silly things like jump­ing up from the table and run­ning away in a panic. How­ever, as your brain reg­isters the new situ­a­tion, the threat sys­tem gets sig­nifi­cantly more strongly ac­ti­vated, send­ing a strong bid for the “let’s run away” ac­tion. As a re­sult of the basal gan­glia re­ceiv­ing that bid, an in­hibitory im­pulse is routed from the basal gan­glia to the sub­sys­tem which was pre­vi­ously sub­mit­ting bids for the “let’s keep eat­ing” ac­tions. This makes the threat sys­tem’s bids even stronger rel­a­tive to the (in­hibited) eat­ing sys­tem’s bids.

Soon the basal gan­glia, which was pre­vi­ously in­hibit­ing the threat sub­sys­tem’s ac­cess to the mo­tor sys­tem while al­low­ing the eat­ing sys­tem ac­cess, with­draws that in­hi­bi­tion and starts in­hibit­ing the eat­ing sys­tem’s ac­cess in­stead. The re­sult is that you jump up from your chair and be­gin to run away. Un­for­tu­nately, this is hope­less since the ve­lo­cirap­tor is faster than you. A few mo­ments later, the ve­lo­cirap­tor’s basal gan­glia gives the rap­tor’s “eat­ing” sub­sys­tem ac­cess to the rap­tor’s mo­tor sys­tem, let­ting it hap­pily munch down its lat­est meal.

But let’s leave ve­lo­cirap­tors be­hind and go back to our origi­nal ex­am­ple with the phone. Sup­pose that you have been try­ing to re­place the habit of look­ing at your phone when bored, to in­stead smil­ing and di­rect­ing your at­ten­tion to pleas­ant sen­sa­tions in your body, and then let­ting your mind wan­der.

Un­til the new habit es­tab­lishes it­self, the two habits will com­pete for con­trol. Fre­quently, the old habit will be stronger, and you will just au­to­mat­i­cally check your phone with­out even re­mem­ber­ing that you were sup­posed to do some­thing differ­ent. For this rea­son, be­hav­ioral change pro­grams may first spend sev­eral weeks just prac­tic­ing notic­ing the situ­a­tions in which you en­gage in the old habit. When you do no­tice what you are about to do, then more goal-di­rected sub­sys­tems may send bids to­wards the “smile and look for nice sen­sa­tions” ac­tion. If this hap­pens and you pay at­ten­tion to your ex­pe­rience, you may no­tice that long-term it ac­tu­ally feels more pleas­ant than look­ing at the phone, re­in­forc­ing the new habit un­til it be­comes preva­lent.

To put this in terms of the sub­agent model, we might dras­ti­cally sim­plify things by say­ing that the neu­ral pat­tern cor­re­spond­ing to the old habit is a sub­agent re­act­ing to a spe­cific sen­sa­tion (bore­dom) in the con­scious­ness workspace: its re­ac­tion is to gen­er­ate an in­ten­tion to look at the phone. At first, you might train the sub­agent re­spon­si­ble for mon­i­tor­ing the con­tents of your con­scious­ness, to out­put mo­ments of in­tro­spec­tive aware­ness high­light­ing when that in­ten­tion ap­pears. That in­tro­spec­tive aware­ness helps alert a goal-di­rected sub­agent to try to trig­ger the new habit in­stead. Grad­u­ally, a neu­ral cir­cuit cor­re­spond­ing to the new habit gets trained up, which starts send­ing its own bids when it de­tects bore­dom. Over time, re­in­force­ment learn­ing in the basal gan­glia starts giv­ing that sub­agent’s bids more weight rel­a­tive to the old habit’s, un­til it no longer needs the goal-di­rected sub­agent’s sup­port in or­der to win.

Now this model helps in­cor­po­rate things like the role of hav­ing a vivid emo­tional mo­ti­va­tion, a sense of hope, or psych­ing your­self up when try­ing to achieve habit change. Do­ing things like imag­in­ing an out­come that you wish the habit to lead to, may ac­ti­vate ad­di­tional sub­sys­tems which care about those kinds of out­comes, caus­ing them to sub­mit ad­di­tional bids in fa­vor of the new habit. The ex­tent to which you suc­ceed at do­ing so, de­pends on the ex­tent to which your mind-sys­tem con­sid­ers it plau­si­ble that the new habit leads to the new out­come. For in­stance, if you imag­ine your ex­er­cise habit mak­ing you strong and healthy, then sub­agents which care about strength and health might ac­ti­vate to the ex­tent that you be­lieve this to be a likely out­come, send­ing bids in fa­vor of the ex­er­cise ac­tion.

On this view, one way for the mind to main­tain co­her­ence and read­just its be­hav­iors, is its abil­ity to re-eval­u­ate old habits in light of which sub­sys­tems get ac­ti­vated when re­flect­ing on the pos­si­ble con­se­quences of new habits. An old habit hav­ing been strongly re­in­forced re­flects that a great deal of ev­i­dence has ac­cu­mu­lated in fa­vor of it be­ing benefi­cial, but the be­hav­ior in ques­tion can still be over­rid­den if enough in­fluen­tial sub­sys­tems weigh in with their eval­u­a­tion that a new be­hav­ior would be more benefi­cial in ex­pec­ta­tion.

Some sub­sys­tems hav­ing con­cerns (e.g. im­me­di­ate sur­vival) which are ranked more highly than oth­ers (e.g. cre­ative ex­plo­ra­tion) means that the de­ci­sion-mak­ing pro­cess ends up car­ry­ing out an im­plicit ex­pected util­ity calcu­la­tion. The strengths of bids sub­mit­ted by differ­ent sys­tems do not just re­flect the prob­a­bil­ity that those sub­sys­tems put on an ac­tion be­ing the most benefi­cial. There are also differ­ent mechanisms giv­ing the bids from differ­ent sub­sys­tems vary­ing amounts of weight, de­pend­ing on how im­por­tant the con­cerns rep­re­sented by that sub­sys­tem hap­pen to be in that situ­a­tion. This ends up do­ing some­thing like weight­ing the prob­a­bil­ities by util­ity, with the kinds of util­ity calcu­la­tions that are cho­sen by evolu­tion and cul­ture in a way to max­i­mize ge­netic fit­ness on av­er­age. Pro­tec­tors, of course, are sub­sys­tems whose bids are weighted par­tic­u­larly strongly, since the sys­tem puts high util­ity on avoid­ing the kinds of out­comes they are try­ing to avoid.

The origi­nal ques­tion which mo­ti­vated this sec­tion was: why are we some­times in­ca­pable of adopt­ing a new habit or aban­don­ing an old one, de­spite know­ing that to be a good idea? And the an­swer is: be­cause we don’t know that such a change would be a good idea. Rather, some sub­sys­tems think that it would be a good idea, but other sub­sys­tems re­main un­con­vinced. Thus the sys­tem’s over­all judg­ment is that the old be­hav­ior should be main­tained.

In­ter­lude: Min­sky on mu­tu­ally bid­ding subagents

I was try­ing to con­cen­trate on a cer­tain prob­lem but was get­ting bored and sleepy. Then I imag­ined that one of my com­peti­tors, Pro­fes­sor Challenger, was about to solve the same prob­lem. An an­gry wish to frus­trate Challenger then kept me work­ing on the prob­lem for a while. The strange thing was, this prob­lem was not of the sort that ever in­ter­ested Challenger.
What makes us use such round­about tech­niques to in­fluence our­selves? Why be so in­di­rect, in­vent­ing mis­rep­re­sen­ta­tions, fan­tasies, and out­right lies? Why can’t we sim­ply tell our­selves to do the things we want to do? [...]
Ap­par­ently, what hap­pened was that my agency for Work ex­ploited Anger to stop Sleep. But why should Work use such a de­vi­ous trick?
To see why we have to be so in­di­rect, con­sider some al­ter­na­tives. If Work could sim­ply turn off Sleep, we’d quickly wear our bod­ies out. If Work could sim­ply switch Anger on, we’d be fight­ing all the time. Direct­ness is too dan­ger­ous. We’d die.
Ex­tinc­tion would be swift for a species that could sim­ply switch off hunger or pain. In­stead, there must be checks and bal­ances. We’d never get through one full day if any agency could seize and hold con­trol over all the rest. This must be why our agen­cies, in or­der to ex­ploit each other’s skills, have to dis­cover such round­about path­ways. All di­rect con­nec­tions must have been re­moved in the course of our evolu­tion.
This must be one rea­son why we use fan­tasies: to provide the miss­ing paths. You may not be able to make your­self an­gry sim­ply by de­cid­ing to be an­gry, but you can still imag­ine ob­jects or situ­a­tions that make you an­gry. In the sce­nario about Pro­fes­sor Challenger, my agency Work ex­ploited a par­tic­u­lar mem­ory to arouse my Anger’s ten­dency to counter Sleep. This is typ­i­cal of the tricks we use for self-con­trol.
Most of our self-con­trol meth­ods pro­ceed un­con­sciously, but we some­times re­sort to con­scious schemes in which we offer re­wards to our­selves: “If I can get this pro­ject done, I’ll have more time for other things.” How­ever, it is not such a sim­ple thing to be able to bribe your­self. To do it suc­cess­fully, you have to dis­cover which men­tal in­cen­tives will ac­tu­ally work on your­self. This means that you—or rather, your agen­cies—have to learn some­thing about one an­other’s dis­po­si­tions. In this re­spect the schemes we use to in­fluence our­selves don’t seem to differ much from those we use to ex­ploit other peo­ple—and, similarly, they of­ten fail. When we try to in­duce our­selves to work by offer­ing our­selves re­wards, we don’t always keep our bar­gains; we then pro­ceed to raise the price or even de­ceive our­selves, much as one per­son may try to con­ceal an unattrac­tive bar­gain from an­other per­son.
Hu­man self-con­trol is no sim­ple skill, but an ever-grow­ing world of ex­per­tise that reaches into ev­ery­thing we do. Why is it that, in the end, so few of our self-in­cen­tive tricks work well? Be­cause, as we have seen, di­rect­ness is too dan­ger­ous. If self-con­trol were easy to ob­tain, we’d end up ac­com­plish­ing noth­ing at all.

-- Marvin Min­sky, The So­ciety of Mind

Akra­sia is sub­agent disagreement

You might feel that the above dis­cus­sion doesn’t still en­tirely re­solve the origi­nal ques­tion. After all, some­times we do man­age to change even strongly con­di­tioned habits pretty quickly. Why is it some­times hard and some­times eas­ier?

Red­grave et al. (2010) dis­cuss two modes of be­hav­ioral con­trol: goal-di­rected ver­sus ha­bit­ual. Goal-di­rected con­trol is a rel­a­tively slow mode of de­ci­sion-mak­ing, where “ac­tion se­lec­tion is de­ter­mined pri­mar­ily by the rel­a­tive util­ity of pre­dicted out­comes”, whereas ha­bit­ual con­trol in­volves more di­rectly con­di­tioned stim­u­lus-re­sponse be­hav­ior. Which kind of sub­sys­tem is in con­trol is com­pli­cated, and de­pends on a va­ri­ety of fac­tors (the fol­low­ing quote has been ed­ited to re­move foot­notes to refer­ences; see the origi­nal for those):

Ex­per­i­men­tally, sev­eral fac­tors have been shown to de­ter­mine whether the agent (an­i­mal or hu­man) op­er­ates in goal-di­rected or ha­bit­ual mode. The first is over-train­ing: here, ini­tial con­trol is largely goal-di­rected, but with con­sis­tent and re­peated train­ing there is a grad­ual shift to stim­u­lus–re­sponse, ha­bit­ual con­trol. Once habits are es­tab­lished, ha­bit­ual re­spond­ing tends to dom­i­nate, es­pe­cially in stress­ful situ­a­tions in which quick re­ac­tions are re­quired. The sec­ond re­lated fac­tor is task pre­dictabil­ity: in the ex­am­ple of driv­ing, talk­ing on a mo­bile phone is fine so long as ev­ery­thing pro­ceeds pre­dictably. How­ever, if some­thing un­ex­pected oc­curs, such as some­one step­ping out into the road, there is an im­me­di­ate switch from ha­bit­ual to goal-di­rected con­trol. Mak­ing this switch takes time and this is one of the rea­sons why sev­eral coun­tries have banned the use of mo­bile phones while driv­ing. The third fac­tor is the type of re­in­force­ment sched­ule: here, fixed-ra­tio sched­ules pro­mote goal-di­rected con­trol as the out­come is con­tin­gent on re­spond­ing (for ex­am­ple, a food pel­let is de­liv­ered af­ter ev­ery n re­sponses). By con­trast, in­ter­val sched­ules (for ex­am­ple, sched­ules in which the first re­sponse fol­low­ing a speci­fied pe­riod is re­warded) fa­cil­i­tate ha­bit­ual re­spond­ing be­cause con­tin­gen­cies be­tween ac­tion and out­come are vari­able. Fi­nally, stress, of­ten in the form of ur­gency, has a pow­er­ful in­fluence over which mode of con­trol is used. The fast, low com­pu­ta­tional re­quire­ments of stim­u­lus–re­sponse pro­cess­ing en­sure that ha­bit­ual con­trol pre­dom­i­nates when cir­cum­stances de­mand rapid re­ac­tions (for ex­am­ple, pul­ling the wrong way in an emer­gency when driv­ing on the op­po­site side of the road). Chronic stress also favours stim­u­lus–re­sponse, ha­bit­ual con­trol. For ex­am­ple, rats ex­posed to chronic stress be­come, in terms of their be­havi­oural re­sponses, in­sen­si­tive to changes in out­come value and re­sis­tant to changes in ac­tion–out­come con­tin­gency. [...]
Although these fac­tors can be seen as pro­mot­ing one form of in­stru­men­tal con­trol over the other, real-world tasks of­ten have mul­ti­ple com­po­nents that must be performed si­mul­ta­neously or in rapid se­quences. Tak­ing again the ex­am­ple of driv­ing, a driver is re­quired to con­tinue steer­ing while chang­ing gear or brak­ing. Dur­ing the first few driv­ing les­sons, when steer­ing is not yet un­der au­to­matic stim­u­lus–re­sponse con­trol, things can go hor­ribly awry when the new driver at­tempts to change gears. By con­trast, an ex­pe­rienced (that is, ‘over-trained’) driver can steer, brake and change gear au­to­mat­i­cally, while hold­ing a con­ver­sa­tion, with only fleet­ing con­tri­bu­tions from the goal-di­rected con­trol sys­tem. This sug­gests that many skills can be de­con­structed into se­quenced com­bi­na­tions of both goal-di­rected and ha­bit­ual con­trol work­ing in con­cert. [...]
Nev­er­the­less, a fun­da­men­tal prob­lem re­mains: at any point in time, which mode should be al­lowed to con­trol which com­po­nent of a task? Daw et al. have used a com­pu­ta­tional ap­proach to ad­dress this prob­lem. Their anal­y­sis was based on the recog­ni­tion that goal-di­rected re­spond­ing is flex­ible but slow and car­ries com­par­a­tively high com­pu­ta­tional costs as op­posed to the fast but in­flex­ible ha­bit­ual mode. They pro­posed a model in which the rel­a­tive un­cer­tainty of pre­dic­tions made by each con­trol sys­tem is tracked. In any situ­a­tion, the con­trol sys­tem with the most ac­cu­rate pre­dic­tions comes to di­rect be­havi­oural out­put.

Note those last sen­tences: be­sides the sub­sys­tems mak­ing their own pre­dic­tions, there might also be a meta-learn­ing sys­tem keep­ing track of which other sub­sys­tems tend to make the most ac­cu­rate pre­dic­tions in each situ­a­tion, giv­ing ex­tra weight to the bids of the sub­sys­tem which has tended to perform the best in that situ­a­tion. We’ll come back to that in fu­ture posts.

This seems com­pat­i­ble with my ex­pe­rience in that, I feel like it’s pos­si­ble for me to change even en­trenched habits rel­a­tively quickly—as­sum­ing that the new habit re­ally is un­am­bigu­ously bet­ter. In that case, while I might for­get and lapse to the old habit a few times, there’s still a rapid feed­back loop which quickly in­di­cates that the goal-di­rected sys­tem is sim­ply right about the new habit be­ing bet­ter.

Or, the be­hav­ior in ques­tion might be suffi­ciently com­plex and I might be suffi­ciently in­ex­pe­rienced at it, that the goal-di­rected (de­fault plan­ning) sub­agent has always mostly re­mained in con­trol of it. In that case change is again easy, since there is no strong ha­bit­ual pat­tern to over­ride.

In con­trast, in cases where it’s hard to es­tab­lish a new be­hav­ior, there tends to be some kind of gen­uine un­cer­tainty:

  • The benefits of the old be­hav­ior have been val­i­dated in the form of di­rect ex­pe­rience (e.g. un­healthy food that tastes good, has in fact tasted good each time), whereas the benefits of the new be­hav­ior come from a less trusted in­for­ma­tion source which is harder to val­i­date (e.g. I’ve read sci­en­tific stud­ies about the long-term health risks of this food).

  • Im­me­di­ate vs. long-term re­wards: the more re­mote the re­wards, the larger the risk that they will for some rea­son never ma­te­ri­al­ize.

  • High vs. low var­i­ance: some­times when I’m bored, look­ing at my phone pro­duces gen­uinely bet­ter re­sults than let­ting my thoughts wan­der. E.g. I might see an in­ter­est­ing ar­ti­cle or dis­cus­sion, which gives me novel ideas or in­sights that I would not oth­er­wise have had. Ba­si­cally look­ing at my phone usu­ally pro­duces worse re­sults than not look­ing at it—but some­times it also pro­duces much bet­ter ones than the al­ter­na­tive.

  • Si­tu­a­tional vari­ables af­fect­ing the value of the be­hav­iors: look­ing at my phone can be a way to es­cape un­com­fortable thoughts or sen­sa­tions, for which pur­pose it’s of­ten ex­cel­lent. This then also tends to re­in­force the be­hav­ior of look­ing at the phone when I’m in the same situ­a­tion oth­er­wise, but with­out un­com­fortable sen­sa­tions that I’d like to es­cape.

When there is sig­nifi­cant un­cer­tainty, the brain seems to fall back to those re­sponses which have worked the best in the past—which seems like a rea­son­able ap­proach, given that in­tel­li­gence in­volves hit­ting tiny tar­gets in a huge search space, so most novel re­sponses are likely to be wrong.

As the above ex­cerpt noted, the ten­dency to fall back to old habits is ex­ac­er­bated dur­ing times of stress. The au­thors at­tribute it to the need to act quickly in stress­ful situ­a­tions, which seems cor­rect—but I would also em­pha­size the fact that nega­tive emo­tions in gen­eral tend to be signs of some­thing be­ing wrong. E.g. El­dar et al. (2016) note that pos­i­tive or nega­tive moods tend to be re­lated to whether things are go­ing bet­ter or worse than ex­pected, and sug­gest that mood is a com­pu­ta­tional rep­re­sen­ta­tion of mo­men­tum, act­ing as a sort of global up­date to our re­ward ex­pec­ta­tions.

For in­stance, if an an­i­mal finds more fruit than it had been ex­pect­ing, that may in­di­cate that spring is com­ing. A shift to a good mood and be­ing “ir­ra­tionally op­ti­mistic” about find­ing fruit even in places where the an­i­mal hasn’t seen fruit in a while, may ac­tu­ally serve as a ra­tio­nal pre-emp­tive up­date to its ex­pec­ta­tions. In a similar way, things go­ing less well than ex­pected may be a sign of some more gen­eral prob­lem, ne­ces­si­tat­ing fewer ex­plo­ra­tory be­hav­iors and less risk-tak­ing, so fal­ling back into be­hav­iors for which there is a higher cer­tainty of them work­ing out.

So to re­peat the sum­mary that I had in the be­gin­ning: we are ca­pa­ble of chang­ing our be­hav­iors on oc­ca­sions when the mind-sys­tem as a whole puts suffi­ciently high prob­a­bil­ity on the new be­hav­ior be­ing bet­ter, when the new be­hav­ior is not be­ing blocked by a par­tic­u­lar highly weighted sub­agent (such as an IFS pro­tec­tor whose bids get a lot of weight) that puts high prob­a­bil­ity on it be­ing bad, and when we have enough slack in our lives for any new be­hav­iors to be eval­u­ated in the first place. Akra­sia is sub­agent dis­agree­ment about what to do.