At 87, Pearl is still able to change his mind
Judea Pearl is a famous researcher, known for Bayesian networks (the standard way of representing Bayesian models), and his statistical formalization of causality. Although he has always been recommended reading here, he’s less of a staple compared to, say, Jaynes. So the need to re-introduce him. My purpose here is to highlight a soothing, unexpected show of rationality on his part.
One year ago I reviewed his last book, The Book of Why, in a failed[1] submission to the ACX book review contest. There I spend a lot of time around what appears to me as a total paradox in a central message of the book, dear to Pearl: that you can’t just use statistics and probabilities to understand causal relationships; you need a causal model, a fundamentally different beast. Yet, at the same time, Pearl shows how to implement a causal model in terms of a standard statistical model.
Before giving me the time to properly raise all my eyebrows, he then sweepingly connects this insight to Everything Everywhere. In particular, he thinks that machine learning is “stuck on rung one”, his own idiomatic expression to say that machine learning algorithms, only combing for correlations in the training data, are stuck at statistics-level reasoning, while causal reasoning resides at higher “rungs” on the “ladder of causation”, which can’t be reached unless you deliberately employ causal techniques.
My rejoinder to this is that, analogously to how a causal model can be re-implemented as a more complex non-causal model[2], a learning algorithm that looks at data that in some ways is saying something about causality, be it because the data contains information-decision-action-outcome units generated by agents, because the learning thing can execute actions itself and reflectively process the information of having done such actions, or because the data contains an abstract description of causality, can surely learn causality. A powerful enough learner ought to be able to cross such levels of quoting.
Thus, I was gleefully surprised to read Pearl expressing this same reasoning in the September cover story of AMSTAT News. Surprised, because his writings, and forever ongoing debates with other causality researchers, begat an image of a very stubborn old man. VERY stubborn. Even when judging him in the right, I deemed him too damn confident and self-aggrandizing. At this point, I could not expect that, after dedicating a whole book to say a thing he had been repeating for 20 years, he could just go on the record and say “Oops”.
He did.
Granted, a partial oops. He says “but”. Still way beyond what I am used to expect from 80 year olds with a sterling hard-nosing track record.
Bits of the interview:
Mackenzie: Can you tell me your first reactions to ChatGPT and GPT-4? Did you find their capabilities surprising?
Pearl: Aside from being impressed, I have had to reconsider my proof that one cannot get any answer to any causal or counterfactual query from observational studies. What I didn’t take into account is the possibility that the text in the training database would itself contain causal information. The programs can simply cite information from the text without experiencing any of the underlying data.
In the next paragraph, he shows the rare skill of not dunking on GPT before proper prompt futzing:
For example, I asked it the questions about the firing squad [from Chapter 1 of The Book of Why], such as what would have happened to the (nowdeceased) prisoner if rifleman 1 had refrained from shooting. At first it goes into side tracks and tells you, for example, “it is dangerous to shoot people.” But if you have time and prompt it correctly, it will get closer to the correct answer: “If soldier 1 refrained from shooting after receiving the signal, the prisoner could still have been killed by soldier 2, assuming he received and acted upon the same signal.” Finally, it gives an A+ answer: “Given the additional information, if each soldier always fires upon receiving a signal and any one soldier’s shot is enough to cause the prisoner’s death, then the prisoner would still be dead if soldier 1 refrained from shooting. This is because soldier 2, following the captain’s signal, would have fired his shot, causing the prisoner’s death. This is an example of ‘overdetermination’ in causation, where an effect (the prisoner’s death) has more than one sufficient cause (either soldier’s shot).”
[...]
Mackenzie: In The Book of Why, we said current AI programs operate at the first level of the ladder of causation, the level of observation or “fitting functions to data.” Has this changed?
Pearl: It has. The ladder restrictions [e.g., level-two queries cannot be answered by level-one data] do not hold anymore because the data is text, and text may contain information on levels two and three.
[...]
Mackenzie: In particular, does reinforcement learning make it possible for a machine to understand level two on the ladder of causation by giving it data on interventions?
Pearl: Yes, that is correct. I would say it’s at level one and three-fourths. Reinforcement learning trains machines on interventions. For example, you can train them on chess. They can decide, after playing many games, that a certain move will give them a higher probability of checkmate than another move. However, they cannot infer from this anything about a third move they haven’t tried. They also cannot combine interventions to infer what will happen if they do both A and B. For that, again, you would need a causal model.
To top off, some AI safety:
Mackenzie: Even AI researchers agree we need ethical guidelines for the use of AI. What guidelines would you recommend?
Pearl: I have to answer this question at two different levels. First, at the level of ChatGPT, it’s already dangerous because it can be misused by dictators or by greedy businesses to do a lot of harm: combining and distorting data, using it to control a segment of the population. That can be done even today with ChatGPT. Some regulation is needed to make sure the technology doesn’t fall to people who will misuse it, even though it’s in the very early stage of development. It’s not general AI yet, but it still can be harmful.
The second danger is when we really have general AI, machines that are a million times more powerful [than humans]. At this point I raise my hands and say we don’t even have the metaphors with which to understand how dangerous it is and what we need to control it.
I used to feel safe about AI. What’s the big deal? We take our chances with teenagers, who think much faster than us. Once in a while we make a mistake and we get a Putin, and the world suffers. But most of the time, education works. But with AI, we are talking about something totally different. Your teenagers are now a hundred million times faster than you, and they have access to a hundred million times larger space of knowledge. Never in history has there been such an acceleration of the speed of evolution. For that reason, we should worry about it, and I don’t know how to even begin to speak about how to control it.
Mackenzie: But didn’t we talk about this in The Book of Why? We discussed the concept of regret, the idea that a machine with a causal model could compare what happened with what would have happened if it took a different course of action. Do you still think regret can equip a machine to make its own ethical judgements?
Pearl: Regret and responsibility will of course be part of AGI and will be implemented eventually using counterfactual logic. Where it will go, I don’t know. No matter how well we program the guards of responsibility for this new species, it might decide it wants to dominate the world on its own. It happened to Homo sapiens. We extinguished all the other forms of human, the Neanderthal and Homo erectus. Imagine what a machine 10 million times smarter could do. It’s unbelievable.
The idea of dominating the world could be one of those local perturbations I talked about. The machine might try it out, decide it’s fun, and pursue it with vigor.
Mackenzie: So are you pessimistic now about giving AIs human- compatible ethics fast enough?
Pearl: You can try to form a committee to regulate it, but I don’t know what that committee will do.
Mackenzie: To conclude the interview, do you have any predictions about what we are going to see in AI in the next year or five years?
Pearl: Do you want to ask me what we are going to see, or what I’d like to see? I’d like to see a shift in emphasis from machine learning to general AI. ChatGPT actually slowed down our progress toward general AI. More and more of our resources will be poured into that direction and not into the correct way of doing AI.
Mackenzie: But maybe that’s a good thing. You said general AI is something to worry about.
Pearl: Here, I am torn. Maybe it’s a blessing that ChatGPT is so stupid and society is so intoxicated with it. So maybe we are safe from the danger of creating the new species I mentioned.
Nice find!
GPT-4 really seems to change the minds of a lot of researchers. Pearl, Hinton, and I think I saw a few others too but can’t remember who they were now.
Hofstadter too!
Yeah. It seems worth reviewing what is was about that innovation in particular that caused people to update, so we can have a better general model of when, “after critical event W happens, they still won’t believe you”, and when it turns out that they will!
I’m wondering if GPT-5 or Gemini would snap people like LeCun out of their complacency. I suspect LeCun has a pretty detailed model of intelligence which implies things like mesa-optimization not being a problem etc. as well as further scaling successes being implausible. Something like Gemini having a good enough world model to do plenty of physical reasoning in a simulation may violate enough of his assumptions that he actually updates.
In the interview, he does not say if he has tried gpt 3 or 4. After witnessing various intellectuals skimping the 20$ and then generalizing whatever gpt3 did to a grand theory of artificial intelligence, I’m not too confident that Pearl ponied up. I say 5:1 he tried gpt4 as well.
How does he think that humans get the causal information in the first place?
If something interests us, we can perform trials. Because our knowledge is integrated with our decisionmaking, we can learn causality that way. What ChatGPT does is pick up both knowledge and decisionmaking by imitation, which is why it can also exhibit causal reasoning without itself necessarily acting agentically during training.
Is this your opinion, or what you think Pearl’s opinion is?
It’s a loose guess at what Pearl’s opinion is. I’m not sure this boundary exists at all.
Ok. My guess is that Pearl would say something more like that we have an innate ability to represent causal models, and only after that follow with what you said. He thinks that having the causal model representation is necessary, that you can’t just look at trials and decisions to make causal inferences, if you don’t have this special causal machinery inside you. (Personally, I disagree this is a good frame.)
Short comment/feedback just to say: This sentence is making one of your main points but is very tricky! - perhaps too long/too many subclauses?
I had to read this sentence a few times to grok the author’s point...
Thanks for sharing! It’s nice to see plasticity, especially for stats, which seems to have more opinionated contributors than other applied maths. Although, it seems this ‘admission’ is not changing his framework, but rather reinterpreting how ML is used to be compatible with his framework.
Pearl’s texts talk about having causal models that use the do(X) operator (e.g. P(Y|do(X))) to signify causal information. Now in LLMs, he sees the text the model is conditioning on as sometimes being do(X) or X. I’m curious what else besides text would count as this. I’m not sure that I recall this correctly but in his third level, you can use purely observational data to infer causality with things like instrumental variables. If I had a ML model that took as input purely numerical input, such as (tar, smoking status, got cancer, and various other health data), should it be able to predict counterfactual results?
I’m uncertain about what the right answer here is, and how Pearl would view this. My guess is a naive ML model would be able to do this provided the data covered the counterfactual cases which is likely for the smoking case. But it would not be as useful for out of sample counterfactual inferences where there is little or no coverage for the interventions and outcomes (e.g. if one of the inputs was ‘location’ it had to predict the effects of smoking on the ISS, where no one smokes). However, if we kept adding more purely observational information about the universe, it feels like we might be able to get a causal model out a transformer-like thing. I’m aware there are some tools that try to extract a DAG from the data as a primitive form of this approach, but is at odds with the Bayesian stats approach of having a DAG first, then checking to see if the DAG holds with the data or vice versa. Please share if you have some references that would be useful.
I think it is true.
If you observe everything in enough detail and your hypothesis space is complete you get counterfactual prediction automatically. Theoretical example: a Solomonoff inductor observes the world, Physical laws satisfy causality, the best prediction algorithm takes that into account, the inductor’s inference favors that algorithm, the algorithm can simulate Physical laws and so produce counterfactuals if needed in the course of its predictions.
If you live in a world where counterfactual thinking is possible and useful to predict the future, then Bayes brings you there.
An interesting look at the question of counterfactuals is the debate between Pearl and Robins on cross-world independence assumptions. It’s relevant because Robins solves the paradox of Pearl’s impossible to verify assumptions by noting that you can always add a mediator in any arrow of a causal model (I’d add, due to locality of Physical laws) and this makes the assumptions verifiable in principle. In other words, by observing the “full video” of a process, instead of just the frames represented by some random variables, you need less out-of-the-hat assumptions to infer counterfactual causal quantities.
I tried to write an explanation, but I realized I still don’t understand the matter enough to go through the details, so I’ll leave you a reference: the last section, “Mediation”, in this Robins interview.
My superficial impression is that the field of causal discovery does not have its shit together. Not to dunk on them; it’s not a law of Nature that what you set out to do will be within your ability. See also “Are there any good, easy-to-understand examples of cases where statistical causal network discovery worked well in practice?”
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?