Key Decision Analysis—a fundamental rationality technique

The technique

This post is sig­nal-boost­ing and recom­mend­ing a strat­egy for im­prov­ing your de­ci­sion mak­ing that I picked up from the en­trepreneur Ivan Ma­zour. He de­scribes the pro­cess here, and pub­lishes his own re­sults ev­ery year on his blog.

In his words...

I be­lieve that life is far too fast-paced for us to be able to make ra­tio­nal, care­fully thought-out, de­ci­sions all the time. This com­pletely con­tra­dicts my math­e­mat­i­cal up­bring­ing and train­ing, but is some­thing I have come to re­al­ise through­out my twen­ties. We need a way of keep­ing up with the con­stant bar­rage of de­ci­sions, even when the in­evitable ‘de­ci­sion fa­tigue’ sets in. The only way to do this, I find, is to act on in­stinct, but this only works if your in­stincts are cor­rect. They can­not be cor­rect all the time, of course, but if we can max­imise the chance of mak­ing the right de­ci­sion by in­stinct, then we have a strat­egy for cop­ing with a com­pli­cated and highly pro­duc­tive life.
To sharpen my in­stincts, I keep a monthly jour­nal of all key de­ci­sions which I make – de­ci­sions that could be truly life chang­ing – and my in­stinc­tive rea­sons for why I made them. I go back only af­ter ex­actly a year has passed, and I note down whether the de­ci­sion was cor­rect, and more im­por­tantly whether my in­stincts were right. At the end of the year, I go over all twelve months worth of notes, and search for any pat­terns amongst all of the right and wrong choices.
This is not a short-term strat­egy, as you can tell. In fact it takes ex­actly two years from the day you start fol­low­ing it, to the time that you can get some use­ful in­sights to sharpen your in­stincts. Keep­ing a di­ary of de­ci­sions has other uses, and there are many ways of get­ting an overview of your life prior to this, but it is only af­ter the two years have passed that a gen­uine clear pat­tern pre­sents it­self.

Some theory

This ac­cords with some ab­stract the­ory about hu­man ra­tio­nal­ity. A perfect-Bayesian ex­pected util­ity max­i­mizer doesn’t start out with an op­ti­mal policy. Rather, its strength is be­ing able to learn from its ex­pe­rience (op­ti­mally), so that it con­verges to­wards the op­ti­mal policy.

Of course, hu­mans have a num­ber of limi­ta­tions stand­ing be­tween us and perfect-de­ci­sion-mak­ing-in-the-limit. Due to com­pu­ta­tional con­straints, perfect Bayesian up­dat­ing is out of reach. But among a num­ber of limi­ta­tions, the first and most fun­da­men­tal con­sid­er­a­tion is “are you learn­ing from your data at all?”.

If the con­se­quences of your de­ci­sions don’t prop­a­gate back to the pro­cess(es) that you use to makes de­ci­sions, then that de­ci­sion pro­cess isn’t go­ing to im­prove.

And I think that, by de­fault, I mostly don’t learn from my own ex­pe­rience, for a cou­ple of rea­sons.

  • Reflec­tion isn’t au­to­matic, I’m likely to make many de­ci­sions, im­por­tant and unim­por­tant, with­out ev­ery go­ing back to check how they turned out, es­pe­cially on long timescales.

  • With hind­sight bias and what­not, I can’t trust my­self to re­mem­ber why I made a de­ci­sion, and how I was think­ing about it at the time, af­ter I’ve seen how it turned out.

  • In gen­eral, each situ­a­tion is treated as an iso­lated in­ci­dent, in­stead ex­am­in­ing at the level of my heuris­tics (i.e. the level of my de­ci­sion mak­ing ap­para­tus).

So I need some pro­cess, that in­volves writ­ing things down, that al­lows me to in­ten­tion­ally im­ple­ment back-prop­a­ga­tion.

Per­sonal experience

I only started log­ging my de­ci­sions a lit­tle more than year ago, and did the anal­y­sis for the end of 2018 this week, so I don’t have that much per­sonal ex­pe­rience to share. I’m shar­ing any­way, be­cause it will be years un­til I have lots of ex­pe­rience with this tech­nique.

That said,

  • I’ve been log­ging very big de­ci­sions (“should I aban­don X pro­ject?”) along with small de­ci­sions (“Some friends (y, z) just asked me if I want to go out to din­ner with them. Should I join them, or keep work­ing?”). In some situ­a­tions, I get feed­back about whether what I did was the right choice or not, pretty much im­me­di­ately, in which case I’ll log that too, so that I can draw out heuris­tics later.

  • I’ve also been log­ging my mis­takes (“I put a lot of effort into set­ting things up so that I could work on the plane, and then my lap­top ran out of bat­tery in the first hour.”).

  • Over­all, I didn’t log enough over the past year, such that my doc­u­ment is sparser than I think it could have been. I av­er­aged 2 to 4 en­tries a month, but I think I could have had 5-10 a month. From look­ing over what I do have, I can feel how hav­ing more en­tries would have been use­ful. So even given the bul­let points above, I think my con­cep­tion of what counts as a “de­ci­sion” was too strict.

  • Re­lat­edly, mak­ing log­ging low-fric­tion seems im­por­tant. This year, I’m go­ing to im­ple­ment this in Roam, us­ing #[[de­ci­sion]] tag, and in­te­grate this into my ex­ist­ing daily /​ weekly re­view sys­tem.

Even given the is­sues I de­scribed above, I found the as­sess­ment ac­tivity to be ex­tremely use­ful. There were some places where I was able to high­light “past Eli was flat-out wrong”, and oth­ers where, hav­ing seen how things turned out, I could out­line nu­anced heuris­tics that take into ac­count the right con­sid­er­a­tions in the right cir­cum­stances.

It also clearly af­firmed two prin­ci­ples /​ ham­ming prob­lems, that had oc­curred to me be­fore, but hadn’t re­ally slapped my in the face. This was helpful for re­al­iz­ing that “my ten­dency to X is pre­emp­tively de­stroy­ing most of the value I might cre­ate”, which is an im­por­tant thing to get to full con­scious at­ten­tion.

Good luck!