An Introduction to Control Theory

Be­hav­ior: The Con­trol of Per­cep­tion by William Pow­ers ap­plies con­trol the­ory to psy­chol­ogy to de­velop a model of hu­man in­tel­li­gence that seems rele­vant to two of LW’s pri­mary in­ter­ests: effec­tive liv­ing for hu­mans and value-pre­serv­ing de­signs for ar­tifi­cial in­tel­li­gence. It’s been dis­cussed on LW pre­vi­ously here, here, and here, as well as men­tioned in Yvain’s roundup of 5 years (and a week) of LW. I’ve found pre­vi­ous dis­cus­sions un­per­sua­sive for two rea­sons: first, they typ­i­cally only have a short in­tro­duc­tion to con­trol the­ory and the me­chan­ics of con­trol sys­tems, mak­ing it not quite ob­vi­ous what spe­cific mod­el­ing tech­niques they have in mind, and sec­ond, they of­ten fail to com­mu­ni­cate the differ­ences be­tween this model and com­pet­ing mod­els of in­tel­li­gence. Even if you’re not in­ter­ested in its ap­pli­ca­tion to psy­chol­ogy, con­trol the­ory is a widely ap­pli­ca­ble math­e­mat­i­cal toolkit whose ba­sics are sim­ple and well worth know­ing.

Be­cause of the length of the ma­te­rial, I’ll split it into three posts. In this post, I’ll first give an in­tro­duc­tion to that sub­ject that’s hope­fully broadly ac­cessible. The next post will ex­plain the model Pow­ers in­tro­duces in his book. In the last post, I’ll provide com­men­tary on the model and what I see as its im­pli­ca­tions, for both LW and AI.

Con­trol the­ory is a cen­tral tool of mod­ern en­g­ineer­ing. Briefly, most in­ter­est­ing things can be mod­eled as dy­nam­i­cal sys­tems, hav­ing both states and rules on how those states change with time. Con­sider the 3D po­si­tion and ve­loc­ity of a ball in a bowl (with fric­tion); six num­bers tell you where the ball is, its speed, and its di­rec­tion of move­ment, and a for­mula tells you how you can pre­dict what those six num­bers will be in the next in­stant given where they are now. Those sys­tems can be char­ac­ter­ized by their at­trac­tors, states that are sta­ble and are the end­point for nearby states. The ball sit­ting mo­tion­less in the bot­tom of the bowl is an at­trac­tor- if it’s already there, it will stay there, and if it’s nearby (re­leas­ing from a cen­time­ter away, for ex­am­ple), it will even­tu­ally end up there. The point of con­trol the­ory is that adding a con­trol to a dy­nam­i­cal sys­tem al­lows you to edit the to­tal sys­tem dy­nam­ics so that the points you want to be sta­ble at­trac­tors are sta­ble at­trac­tors.

Let’s flesh out that sketch with an ex­am­ple. Sup­pose you want to keep a house within a spe­cific tem­per­a­ture range. You have a sen­sor of the cur­rent tem­per­a­ture, a heater, and a cooler. A ther­mo­stat takes the sen­sor’s out­put, com­pares it to the de­sired tem­per­a­ture range, and turns the heater on if the sensed tem­per­a­ture is be­low the de­sired tem­per­a­ture range, and turns if off if the sensed tem­per­a­ture is above the min­i­mum of that range, and does the re­verse with the cooler (turn­ing it on if the sen­sor is above the de­sired max, and turn­ing it off if it’s be­low).

Most in­ter­est­ing con­trol sys­tems are have a finer range of con­trol val­ues- in­stead of sim­ply flip­ping an on or off switch, a car’s cruise con­trol can smoothly vary the amount of gas or brake that’s be­ing ap­plied. A sim­ple way to make a con­trol sys­tem is to take the differ­ence be­tween the de­sired speed and ac­tual speed, mul­ti­ply it by some fac­tor to go from units of dis­tance per time to an­gle of pedal, and ad­just the po­si­tion of the ped­als ac­cord­ingly. (If the func­tion is lin­ear, it’s called a lin­ear con­trol­ler.)

Let’s in­tro­duce more of the tech­ni­cal vo­cab­u­lary. The thing we’re mea­sur­ing is the in­put (to the con­trol­ler), the level we want it to be at is the refer­ence, the differ­ence be­tween those is the er­ror, and the ad­just­ment the con­trol sys­tem makes is the out­put or feed­back (some­times we’ll talk about the ac­tu­a­tor as the phys­i­cal means by which the con­trol­ler emits its out­put). None of them have to be sin­gle vari­ables- they can be vec­tors, which al­low us to de­scribe ar­bi­trar­ily com­pli­cated sys­tems. (The six num­bers that ex­press the po­si­tion and ve­loc­ity of a ball, each in three di­men­sions, are an ex­am­ple of an in­put vec­tor.) I’ll of­ten use the noun state to re­fer to the, well, state of the sys­tem, and ‘points in state-space’ refers to those states as vec­tors. There’s also a pos­si­ble con­fu­sion in that the plant (the sys­tem be­ing con­trol­led) and the con­trol­ler are mir­rored- the con­trol­ler’s out­put is the plant’s in­put, and the plant’s out­put is the con­trol­ler’s in­put.

Con­trol sys­tems nat­u­rally lend them­selves to di­a­grams: here’s the block di­a­gram from the ther­mo­stat and cruise con­trol:

In a block di­a­gram, each block is some func­tion of its in­puts, and the ar­rows show what af­fects what. Mov­ing left to right, the refer­ence is the tem­per­a­ture you’ve set, the cur­rent tem­per­a­ture is sub­tracted (that’s the point of the plus and minus sign), and the er­ror goes into the yel­low box which rep­re­sents the func­tion that con­verts from the er­ror to the effort put into al­ter­ing the sys­tem. That’s the con­trol­ler out­put ar­row that goes into the green box (the house), which rep­re­sents the ex­ter­nal sys­tem. This is also a func­tional block, be­cause it takes the con­trol­ler out­put and any dis­tur­bances (of­ten rep­re­sented as an­other ar­row point­ing in from the top) and con­verts them into the sys­tem tem­per­a­ture. The ar­row lead­ing out of the house points both to the right- to re­mind you that this is the tem­per­a­ture you’re liv­ing with- and back into the ther­mo­cou­ple, the sen­sor that mea­sures the tem­per­a­ture to com­pare with the refer­ence, and now we’ve finished our feed­back loop.

Now that we have a math­e­mat­i­cal lan­guage for mod­el­ing lots of differ­ent sys­tems, we can ab­stract away the speci­fics and prove prop­er­ties about how those sys­tems will be­have given var­i­ous con­trol­lers (i.e. feed­back func­tions). Feed­back func­tions con­vert the in­put and refer­ence to the out­put, and are the math­e­mat­i­cal ab­strac­tion of a phys­i­cal con­trol­ler. They can be ar­bi­trary func­tions, but most of the math­e­mat­i­cal ed­ifice of con­trol the­ory as­sumes that ev­ery­thing is con­tin­u­ous (but not nec­es­sar­ily lin­ear). If you know the dy­nam­ics of a sys­tem, you can op­ti­mize your con­trol func­tion to match the sys­tem and be guaran­teed to con­verge to the refer­ence with a par­tic­u­lar time pro­file. Rather than go deeply into the math, I’ll dis­cuss a few con­cepts that have tech­ni­cal mean­ings in con­trol the­ory that are use­ful to think about.

First is con­ver­gence: the sys­tem out­put will even­tu­ally match the refer­ence. This means that any er­rors that get in­tro­duced into the sys­tem are tran­sient (tem­po­rary), and ideally we know the time pro­file of how large those er­rors will be as time pro­gresses. A com­mon goal here is ex­po­nen­tial con­ver­gence, which means that the er­ror de­creases with a rate pro­por­tional to its size. (If the tem­per­a­ture is off by 2 de­grees, the rate at which the er­ror is de­creas­ing is twice that of the rate when the tem­per­a­ture is off by 1 de­gree.) A sim­ple lin­ear con­trol­ler will, for sim­ple state dy­nam­ics, ac­com­plish ex­po­nen­tial con­ver­gence. If your sys­tem doesn’t con­verge, then you are not suc­cess­fully con­trol­ling it, and if your refer­ence changes un­pre­dictably at a rate faster than your sys­tem can con­verge, then you are not go­ing to be able to match your refer­ence closely.

Se­cond is equil­ibrium: a point when the forces are bal­anced. If the sys­tem is at an equil­ibrium state, then noth­ing will change. This is typ­i­cally dis­cussed along with steady state er­ror: imag­ine that my house gets heated by the sun at a rate of 1° an hour, and rather than a stan­dard air con­di­tioner that’s on or off I have a ‘dim­mer switch’ for my AC. If my con­trol­ler sets the AC rate at the differ­ence be­tween the refer­ence tem­per­a­ture and the cur­rent tem­per­a­ture per hour, then when the house is at 30° and I want it to be at 23° it’ll try to re­duce the tem­per­a­ture by 7° an hour, but when the house is at 24° and I want it to be at 23° it will try to re­duce the tem­per­a­ture by 1° an hour, which can­cels the effect of the sun, and so the house is at equil­ibrium at 24°. (Most real con­trol­lers have an in­te­gra­tor in or­der to coun­ter­act this effect.)

Third is sta­bil­ity: even when we know a point is an equil­ibrium, we want to know about the be­hav­ior in the neigh­bor­hood of that point. A sta­ble equil­ibrium is one where a dis­tur­bance will be cor­rected; an un­sta­ble equil­ibrium is one where a dis­tur­bance will be am­plified. Imag­ine a pen­du­lum with the bob bal­anced at the bot­tom- tap it and it’ll even­tu­ally be bal­anced at the bot­tom again, be­cause that’s a sta­ble equil­ibrium (also called an at­trac­tor). Now imag­ine the bob bal­anced at the top- tap it, and it’ll race away from that point, un­likely to re­turn, be­cause that’s an un­sta­ble equil­ibrium (also called a re­pul­sor).

Sta­bil­ity has a sec­ond mean­ing in con­trol the­ory: a con­trol­ler that ap­plies too much feed­back will cause the sys­tem to go from a smaller pos­i­tive er­ror to a mod­er­ate nega­tive er­ror, and then again too much feed­back will be ap­plied, and the sys­tem will go from a mod­er­ate nega­tive er­ror to a huge pos­i­tive er­ror. Imag­ine a shower with a de­lay: you turn the tem­per­a­ture knob and five sec­onds later the tem­per­a­ture of the wa­ter com­ing out of the show­er­head changes. If you re­act too strongly or too quickly, then you’ll over­re­act, and the wa­ter that started out too hot will cor­rect to be­ing too cold by the time you’ve stopped turn­ing the knob away from heat. That an overex­u­ber­ant nega­tive feed­back con­trol­ler can still lead to ex­plo­sions is one of the in­ter­est­ing re­sults of con­trol the­ory, as is that mak­ing small, grad­ual, pro­por­tionate changes to the cur­rent state can be as effec­tive as mem­ory /​ im­ple­ment­ing a de­lay in your con­trol­ler. Some­times, you achieve bet­ter con­trol ex­actly be­cause you ap­plied less con­trol effort.

It’s also worth men­tion­ing here that ba­si­cally all real con­trol­lers (even am­plifiers!) are nega­tive feed­back con­trol­lers- pos­i­tive feed­back leads to ex­plo­sions (liter­ally), be­cause “pos­i­tive feed­back” in this con­text means “push­ing the sys­tem in the di­rec­tion of the er­ror” rather than its mean­ing in psy­chol­ogy.

So what’s the point of con­trol sys­tems? If we have a way of push the states of a sys­tem, we can effec­tively ad­just the sys­tem dy­nam­ics to make any state that we want a sta­ble state. If we put a mo­tor at the joint of a pen­du­lum, we can ad­just the ac­cel­er­a­tion of the bob so that it has an equil­ibrium at one ra­dian away from ver­ti­cal, rather than zero ra­di­ans away from ver­ti­cal. If we put a ther­mo­stat in a house, we can ad­just the tem­per­a­ture changes of the house so that it re­turns to a com­fortable range, in­stead of what­ever the tem­per­a­ture is out­side. (The point of con­trol the­ory is to un­der­stand how the sys­tems work, so we can make sure that our con­trol sys­tems do what we want them to do!)

Why are con­trol sys­tems in­ter­est­ing? Three pri­mary rea­sons:

  1. They’re prac­ti­cal. Con­trol sys­tems are all over the place, from ther­mostats to cars to planes to satel­lites.

  2. They can be adap­tive. A hi­er­ar­chi­cal con­trol sys­tem has a con­trol loop which de­ter­mines the pa­ram­e­ters used in an­other con­trol loop, and a nat­u­ral ap­pli­ca­tion is adap­tive con­trol sys­tems. When you launch a satel­lite, you might not have pre­cise mea­sure­ments of its ro­ta­tional in­er­tia, or you might ex­pect that to change as it uses its fuel over its life­time. One con­trol sys­tem could ob­serve how the satel­lite moves in re­sponse to its thrusters, and ad­just the pa­ram­e­ters of a ro­ta­tional in­er­tia model to cor­rect er­rors to match the model to the ob­ser­va­tions. A sec­ond con­trol sys­tem could use the in­er­tia model cre­ated by the first con­trol sys­tem to de­ter­mine how to use the thrusters to ad­just the satel­lite’s al­ign­ment to match the de­sired ro­ta­tion.

  3. They give con­crete math­e­mat­i­cal mod­els of how sim­ple sig­nal pro­cess­ing can cre­ate ‘in­ten­tional’ be­hav­ior, and of what it looks like to be in­ten­tional with­out an ex­plicit model of re­al­ity. A cen­trifu­gal gov­er­nor is not an agent in the LW sense of the term, but it is an agent in an­other sense of the term—an en­tity that performs ac­tions on be­half of an­other. The gov­er­nor is just some pieces of metal, it doesn’t have a mind, it’s not viewed with moral con­cern, it doesn’t have goals as hu­mans think of them, and it doesn’t have even a rudi­men­tary in­ter­nal model of the dy­nam­i­cal sys­tem it’s con­trol­ling, and it still gets the job done. Con­trol­lers seem like a class of en­tities that are best mod­eled by in­ten­tion­al­ity, in that they al­ter the state of their ex­ter­nal en­vi­ron­ment to match their de­sired in­ter­nal en­vi­ron­ment based on their per­cep­tions, and can do so in ar­bi­trar­ily com­pli­cated and pow­er­ful ways, but while they do steer the fu­ture they don’t seem to be cross-do­main and they don’t look like an­thro­po­mor­phic mod­els of “hu­man in­tel­li­gence.”

Next, B:CP on the use of con­trol the­ory in psy­chol­ogy.