Subspace optima

The term “global op­ti­mum” and “lo­cal op­ti­mum” have come from math­e­mat­i­cal ter­minol­ogy and en­tered daily lan­guage. They are use­ful ways of think­ing in ev­ery day life. Another use­ful con­cept, which I don’t hear peo­ple talk about much is “sub­space op­ti­mum”: A point max­i­mizes a func­tion not in the whole space, but in a sub­space. You have to move along a differ­ent di­men­sion than those of the sub­space in or­der to im­prove. A sub­space op­ti­mum doesn’t have to be a lo­cal op­ti­mum ei­ther, be­cause even a small change along the new di­men­sion might yield im­prove­ments. If you’re in a sub­space op­ti­mum, this re­quires a differ­ent at­ti­tude to get to a global op­ti­mum, than if you’re in a lo­cal op­ti­mum, which makes me think it’s good for the term to be part of ev­ery day lan­guage.

  • When you’re in a lo­cal op­ti­mum, you have to do some­thing quite differ­ent from what you’re do­ing to im­prove.

  • When you’re in a sub­space op­ti­mum, you have to no­tice di­men­sions along which you could be do­ing things differ­ently that you didn’t even no­tice be­fore, but small changes along those new di­men­sions might already help. You’re ap­ply­ing con­straints to your­self that you could let go.

Re­gard­ing how it looks sub­jec­tively:

  • The phrase: “am I in a lo­cal op­ti­mum?” gen­er­ates cu­ri­os­ity about whether you maybe should un­der­take a quite differ­ent plan from the one you’re tak­ing now. (Should I do a differ­ent pro­ject, rather than make lo­cal changes to the pro­ject I’m tak­ing?)

  • The phrase: “am I in a sub­space op­ti­mum?” gen­er­ates cu­ri­os­ity about whether you maybe are not notic­ing (pos­si­bly small) changes you could be mak­ing across di­men­sions you haven’t been con­sid­er­ing. (Should I op­ti­mize/​ad­just the way I’m do­ing my pro­ject across differ­ent di­men­sions/​vari­ables than the ones I’ve been op­ti­miz­ing over so far?)

My im­pres­sion is that some­what of­ten when peo­ple in­for­mally use the term lo­cal op­ti­mum, they are in fact talk­ing about a sub­space op­ti­mum.

Bonus for the the­o­ret­i­cally in­clined: A lo­cal sub­space op­ti­mum is one where you can im­prove by tem­porar­ily do­ing things differ­ently along di­men­sion X, mov­ing around in a big­ger space, while even­tu­ally end­ing up on a differ­ent, bet­ter, point in the same sub­space.