Sequence Announcement: Applied Causal Inference

Ap­plied Causal In­fer­ence for Ob­ser­va­tional Research

This se­quence is an in­tro­duc­tion to ba­sic causal in­fer­ence. It was origi­nally writ­ten as aux­iliary notes for a course in Epi­demiol­ogy, but it is rele­vant to al­most any kind of ap­plied statis­ti­cal re­search, in­clud­ing econo­met­rics, so­ciol­ogy, psy­chol­ogy, poli­ti­cal sci­ence etc. I would not be sur­prised if you guys find a lot of er­rors, and I would be very grate­ful if you point them out in the com­ments. This will help me im­prove my course notes and po­ten­tially help me im­prove my un­der­stand­ing of the ma­te­rial.

For math­e­mat­i­cally in­clined read­ers, I recom­mend skip­ping this se­quence and in­stead read­ing Pearl’s book on Causal­ity. There is also a lot of good ma­te­rial on causal graphs on Less Wrong it­self. Also, note that my the­sis ad­vi­sor is writ­ing a book that cov­ers the same ma­te­rial in more de­tail, the first two parts are available for free at his web­site.

Pearl’s book, Miguel’s book and Eliezer’s writ­ings are all more rigor­ous and pre­cise than my se­quence. This is partly be­cause I have a differ­ent goal: Pearl and Eliezer are writ­ing for math­e­mat­i­ci­ans and the­o­rists who may be in­ter­ested in con­tribut­ing to the the­ory. In­stead, I am writ­ing for con­sumers of sci­ence who want to un­der­stand cor­re­la­tion stud­ies from the per­spec­tive of a more rigor­ous episte­mol­ogy.

I will use Epi­demiolog­i­cal/​Coun­ter­fac­tual no­ta­tion rather than Pearl’s no­ta­tion. I apol­o­gize if this is con­fus­ing. Th­ese two ap­proaches re­fer to the same math­e­mat­i­cal ob­jects, it is just a differ­ent no­ta­tion. Whereas Pearl would use the “Do-Oper­a­tor” E[Y|do(a)], I use coun­ter­fac­tual vari­ables E[Ya]. In­stead of us­ing Pearl’s “Do-Calcu­lus” for iden­ti­fi­ca­tion, I use Robins’ G-For­mula, which will give the same re­sults.

For all ap­pli­ca­tions, I will use the let­ter “A” to rep­re­sent “treat­ment” or “ex­po­sure” (the thing we want to es­ti­mate the effect of), Y to rep­re­sent the out­come, L to rep­re­sent any mea­sured con­founders, and U to rep­re­sent any un­mea­sured con­founders.

Out­line of Se­quence:

I hope to pub­lish one post ev­ery week. I have rough drafts for the fol­low­ing eight sec­tions, and will keep up­dat­ing this out­line with links as the se­quence de­vel­ops:

Part 0: Se­quence An­nounce­ment /​ In­tro­duc­tion (This post)

Part 1: Ba­sic Ter­minol­ogy and the As­sump­tions of Causal Inference

Part 2: Graph­i­cal Models

Part 3: Us­ing Causal Graphs to Un­der­stand Bias

Part 4: Time-Depen­dent Exposures

Part 5: The G-Formula

Part 6: In­verse Prob­a­bil­ity Weighting

Part 7: G-Es­ti­ma­tion of Struc­tural Nested Models and In­stru­men­tal Variables

Part 8: Sin­gle World In­ter­ven­tion Graphs, Cross-World Coun­ter­fac­tu­als and Me­di­a­tion Analysis

In­tro­duc­tion: Why Causal In­fer­ence?

The goal of ap­plied statis­ti­cal re­search is al­most always to learn about causal effects. How­ever, causal in­fer­ence from ob­ser­va­tional is hard, to the ex­tent that it is usu­ally not even pos­si­ble with­out strong, al­most heroic as­sump­tions. Be­cause of the in­her­ent difficulty of the task, many old-school in­ves­ti­ga­tors were trained to avoid mak­ing causal claims. Words like “cause” and “effect” were ban­ished from po­lite com­pany, and the slo­gan “cor­re­la­tion does not im­ply cau­sa­tion” be­came an ar­ti­cle of faith which, when said loudly enough, seem­ingly ab­solved the in­ves­ti­ga­tors from the sin of mak­ing causal claims.

How­ever, read­ers were not fooled: They always un­der­stood that epi­demiologic pa­pers were mak­ing causal claims. Of course they were mak­ing causal claims; why else would any­body be in­ter­ested in a pa­per about the cor­re­la­tion be­tween two vari­ables? For ex­am­ple, why would any­body want to know about the cor­re­la­tion be­tween eat­ing nuts and longevity, un­less they were won­der­ing if eat­ing nuts would cause them to live longer?

When read­ers in­ter­preted these pa­pers causally, were they sim­ply ig­nor­ing the caveats, draw­ing con­clu­sions that were not in­tended by the au­thors? Of course they weren’t. The dis­cus­sion sec­tions of epi­demiologic ar­ti­cles are full of “policy im­pli­ca­tions” and spec­u­la­tions about biolog­i­cal path­ways that are com­pletely con­tin­gent on in­ter­pret­ing the find­ings causally. Quite clearly, no mat­ter how hard the in­ves­ti­ga­tors tried to deny it, they were mak­ing causal claims. How­ever, they were us­ing method­ol­ogy that was not de­signed for causal ques­tions, and did not have a clear lan­guage for rea­son­ing about where the un­cer­tainty about causal claims comes from.

This was not sus­tain­able, and in­evitably led to a crisis of con­fi­dence, which cul­mi­nated when some high-pro­file ran­dom­ized tri­als showed com­pletely differ­ent re­sults from the pre­ced­ing ob­ser­va­tional stud­ies. In one par­tic­u­lar case, when the Women’s Health Ini­ti­a­tive trial showed that post-menopausal hor­mone re­place­ment ther­apy in­creases the risk of car­dio­vas­cu­lar dis­ease, the differ­ence was so dra­matic that many thought-lead­ers in clini­cal medicine com­pletely aban­doned the idea of in­fer­ring causal re­la­tion­ships from ob­ser­va­tional data.

It is im­por­tant to rec­og­nize that the prob­lem was not that the re­sults were wrong. The prob­lem was that there was un­cer­tainty that was not taken se­ri­ously by the in­ves­ti­ga­tors. A ra­tio­nal per­son who wants to learn about the world will be will­ing to ac­cept that stud­ies have er­rors of mar­gin, but only as long as the in­ves­ti­ga­tors make a good-faith effort to ex­am­ine what the sources of er­ror are, and com­mu­ni­cate clearly about this un­cer­tainty to their read­ers. Old-school epi­demiol­ogy failed at this. We are not go­ing to make the same mis­take. In­stead, we are go­ing to de­velop a clear, pre­cise lan­guage for rea­son­ing about un­cer­tainty and bias.

In this con­text, we are go­ing to talk about two sources of un­cer­tainty – “statis­ti­cal” un­cer­tainty and “epi­demiolog­i­cal” un­cer­tainty.

We are go­ing to use the word “Statis­tics” to re­fer to the the­ory of how we can learn about cor­re­la­tions from limited sam­ples. For statis­ti­ci­ans, the pri­mary source of un­cer­tainty is sam­pling vari­abil­ity. Statis­ti­ci­ans are very good at ac­count­ing for this type of un­cer­tainty: Con­cepts such as “stan­dard er­rors”, “p-val­ues” and “con­fi­dence in­ter­vals” are all at­tempts at quan­tify­ing and com­mu­ni­cat­ing the ex­tent of un­cer­tainty that re­sults from sam­pling vari­abil­ity.

The old school of epi­demiol­ogy would tell you to stop af­ter you had found the cor­re­la­tions and ac­counted for the sam­pling vari­abil­ity. They be­lieved go­ing fur­ther was im­pos­si­ble. How­ever, cor­re­la­tions are sim­ply not in­ter­est­ing. If you truly be­lieved that cor­re­la­tions tell you noth­ing about cau­sa­tion, there would be no point in do­ing the study.

There­fore, we are go­ing to use the terms “Epi­demiol­ogy” or “Causal In­fer­ence” to re­fer to the next stage in the pro­cess: Learn­ing about cau­sa­tion from cor­re­la­tions. This is a much harder prob­lem, with many ad­di­tional sources of un­cer­tainty, in­clud­ing con­found­ing and se­lec­tion bias. How­ever, rec­og­niz­ing that the prob­lem is hard does not mean that you shouldn’t try, it just means that you have to be care­ful. As we will see, it is pos­si­ble to rea­son rigor­ously about whether cor­re­la­tion re­ally does im­ply cau­sa­tion in your par­tic­u­lar study: You will just need a pre­cise lan­guage. The goal of this se­quence is sim­ply to give you such a lan­guage.

In or­der to teach you the logic of this lan­guage, we are go­ing to make sev­eral con­tro­ver­sial state­ments such as «The only way to es­ti­mate a causal effect is to run a ran­dom­ized con­trol­led trial» . You may not be will­ing to be­lieve this at first, but in or­der to un­der­stand the logic of causal in­fer­ence, it is nec­es­sary that you are at least will­ing to sus­pend your dis­be­lief and ac­cept it as true within the course.

It is im­por­tant to note that we are not just say­ing this to try to con­vince you to give up on ob­ser­va­tional stud­ies in fa­vor of ran­dom­ized con­trol­led tri­als. We are mak­ing this point be­cause un­der­stand­ing it is nec­es­sary in or­der to ap­pre­ci­ate what it means to con­trol for con­found­ing: It is not pos­si­ble to give a co­her­ent mean­ing to the word “con­found­ing” un­less one is try­ing to de­ter­mine whether it is rea­son­able to model the data as if it came from a com­plex ran­dom­ized trial run by na­ture.

--

When we say that causal in­fer­ence is hard, what we mean by this is not that it is difficult to learn the ba­sics con­cepts of the the­ory. What we mean is that even if you fully un­der­stand ev­ery­thing that has ever been writ­ten about causal in­fer­ence, it is go­ing to be very hard to in­fer a causal re­la­tion­ship from ob­ser­va­tional data, and that there will always be un­cer­tainty about the re­sults. This is why this se­quence is not go­ing to be a work­shop that teaches you how to ap­ply magic causal method­ol­ogy. What we are in­ter­ested in, is de­vel­op­ing your abil­ity to rea­son hon­estly about where un­cer­tainty and bias comes from, so that you can com­mu­ni­cate this to the read­ers of your stud­ies. What we want to teach you about, is the episte­mol­ogy that un­der­lies epi­demiolog­i­cal and statis­ti­cal re­search with ob­ser­va­tional data.

In­sist­ing on only us­ing ran­dom­ized tri­als may seem at­trac­tive to a purist, it does not take much imag­i­na­tion to see that there are situ­a­tions where it is im­por­tant to pre­dict the con­se­quences of an ac­tion, but where it is not pos­si­ble to run a trial. In such situ­a­tions, there may be Bayesian ev­i­dence to be found in na­ture. This ev­i­dence comes in the form of cor­re­la­tions in ob­ser­va­tional data. When we are stuck with this type of ev­i­dence, it is im­por­tant that we have a clear frame­work for as­sess­ing the strength of the ev­i­dence.

--

I am pub­lish­ing Part 1 of the se­quence at the same time as this in­tro­duc­tion. I would be very in­ter­ested in hear­ing feed­back, par­tic­u­larly about whether peo­ple feel this has already been cov­ered in suffi­cient de­tail on Less Wrong. If there is no de­mand, there won’t re­ally be any point in trans­form­ing the rest of my course notes to a Less Wrong for­mat.

Thanks to ev­ery­one who had a look at this be­fore I pub­lished, in­clud­ing pa­per-ma­chine and Vika, Janos, Eloise and Sam from the Bos­ton Meetup group.