Criticism of Pearl's theory of causality

Question

In the year 2000, Judea Pearl published Causality. What controversies surround this work? What are its major criticisms?

Answer 1

Some authors dislike Pearl's focus on the directed acyclic graph (DAG) as the way in which to view causality. Pearl essentially argues that any causal system can be considered as a non-parametric structural equation model (NPSEM), in which the value of each node is taken as a function of its parents and some individual error term; the error terms between different nodes may in general be correlated, to represent common causes.

Cartwright's book Hunting Causes and Using Them, for example, gives an example involving a car engine, which she claims cannot be modelled in the NPSEM framework. Pearl disputes this in his review of Cartwright's book.

Others caution that the use of DAGs can be misleading, in that the arrows lend an apparent authority to a chosen model as having causal implications, when this may not be the case at all. See Dawid's Beware of the DAG. For example, the three DAGs $A \rightarrow B \rightarrow C$, $A \leftarrow B \rightarrow C$ and $A \leftarrow B \leftarrow C$ all induce the same probabilistic model under Pearl's d-separation criterion, which is that A is independent of C given B. They are therefore indistinguishable based upon observational data.

However they have quite different causal interpretations, so if we wish to learn about the causal relationships here we would need more than simply observational data, whether that be the results of interventional experiments, prior information about the system, or something else.

Answer 2

I think this framework has a lot of trouble with general equilibrium effects or Stable Unit Treatment Value Assumption violations. In that case, the "untreated" observations no longer provide the desired counterfactual in a meaningful way. Massive job training programs that shift the entire wage distribution are one example. The counterfactual may not even be well-defined in some cases. In Morgan and Winship's Counterfactuals and Causal Models, they give an example of the claim that the 2000 election would have gone in favor of Al Gore if felons and ex-felons had been allowed to vote. They point out that the counterfactual world would have very different candidates and issues, so that you cannot characterize the alternative causal state. The ceteris paribus effect would not be the policy relevant parameter here.

Answer 3

Counterfactual formal causal reasoning of the form motivated by Pearl is ill-suited to the analysis of complex dynamic causal systems.¹

Disclaimer 1: I am a fan of Pearl's framework, and had the privilege of being taught counterfactual formal causal inference by two of his early exponents (Hernán and Robins).

Disclaimer 2: Levins was my mentor when I was a doctoral student, and I have published using his methods.

The counterfactual theory of causality, and the counterfactual formal causal inference/reasoning built atop of it, are profoundly useful for reasoning through both the strengths and weaknesses of causal inference based on specific combinations of study design and analyses. However, to my mind, the counterfactual theory is a theory of terminal causal narratives: $A$, and $L$, and $U$, (and maybe $V$ or $E$) happened, and then they caused $\pmb{Y}$ to happen (or not). However, the counterfactual theory of causality does not appear to describe or infer the behavior of complex causal systems (i.e. networks in which every variable is either directly or indirectly the cause of every other variable at some future time), and is thus not a theory of cyclic causal narratives.

I would raise as a counterexample of a formal causal reasoning system Levins' qualitative loop analysis, which, like Pearl's work with DAGs also hearkens back to Wright's path analysis, but employs signed digraphs in a different causal formalism (in fact, one obvious distinction is that qualitative loop analysis employs signed digraphs which are cyclic, not acyclic), to describe the behavior of such causal systems under different kinds of perturbation.

The questions posed and answered by Levins' method (and subsequent elaborations on it) include:

• How does the level of each variable in a complex system respond to press perturbation at one or more variables in the system?
• How does the life expectancy/turnover of each variable in a complex system respond to press perturbation at one or more variables in the system?
• Does variance induced by system perturbation (at specific variables) tend to diffuse across the system, or sink into very few variables in the system?
• Where do (Lyapunov) stability and instability emerge in the system?
• Where does system behavior depend on either ontological or epistemic uncertainty regarding the existence or magnitude of specific direct causal relationships comprising the system?
• What are the signs of expected bivariate correlations (or correspondences) between any variable pairs given a press perturbation at one or more variables in the system?

(Because most of Levins' loop analysis is a purely deductive method—although, see Dambacher's extension—only the bolded question is directly statistical.)

These questions are different questions than the ones posed and answered in the counterfactual formal causal inference championed by Pearl. I have even had difficulty finding examples of counterfactual formal causal inference applied in the context of stochastic processes and autoregressive models (e.g., dynamic models including $Y_{t}$ as a function of $Y_{<t}$), although this may be more due to my lack of familiarity with the intersections of Bayesian probabilistic causal graphs and Pearl's work, than due to a specific deficiency in the latter.

Aside: Sugihara's empirical dynamic modeling (see tutorial by Chaing, et al.) elaboration on state space reconstructions, likewise provides an alternative perspective to counterfactual formal causal reasoning, also from the world of complex causal systems.

Video of Sugihara and friends blowing your mind and mine by recovering the topology of 3D complex system from a 1D time series.

¹ A point similar to something Spirtes pointed out quite a while ago.

References

Chang, C.-W., Ushio, M., & Hseih, C. (2017). Empirical Dynamic Modeling for Beginners. Ecological Research, 32(6), 785–796.

Dambacher, J. M., Li, H. W., & Rossignol, P. A. (2003). Qualitative predictions in model ecosystems. Ecological Modelling, 161, 79 /93.

Dambacher, J. M., Levins, R., & Rossignol, P. A. (2005). Life expectancy change in perturbed communities: Derivation and qualitative analysis. Mathematical Biosciences, 197, 1–14.

Levins, R. (1974). The Qualitative Analysis of Partially Specified Systems. Annals of the New York Academy of Sciences, 231, 123–138.

Puccia, C. J., & Levins, R. (1986). Qualitative Modeling of Complex Systems: An Introduction to Loop Analysis and Time Averaging. Harvard University Press.

Spirtes, P. (1995). Directed Cyclic Graphical Representations of Feedback. In P. Besnard & S. Hanks (Eds.), Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann Publishers, Inc.

Sugihara, G., May, R., Ye, H., Hsieh, C., Deyle, E., Fogarty, M., & Munch, S. (2012). Detecting Causality in Complex Ecosystems. Science, 338, 496–500.

Wright, S. (1934). The Method of Path Coefficients. The Annals of Mathematical Statistics, 5(3), 161–215.

Answer 4

The most important criticism of Pearl's system is, from my perspective, that it has not yielded any practical, empirical advances anywhere it has been used. Given how long it has been around, there's no reason to think it will ever be a practical tool. This indicates that it can be used for some theoretical and perhaps didactic purposes, but a practical researcher will gain little from studying it.

Answer 5

Reading answers and comments I feel the opportunity to add something.

The accepted answer, by rje42, is focused on DAG’s and non-parametric systems; strongly related concepts. Now, capabilities and limitations of these tools can be argued, however we have to say that linear SEMs are part of the Theory presented in Pearl manual (2000 or 2009). Pearl underscores limitations of linear systems but they are part of the Theory presented.

The comment of Paul seems crucial to me: “The whole point of Pearl's work is that different causal models can generate the same probabilistic model and hence the same-looking data. So it's not enough to have a probabilistic model, you have to base your analysis and causal interpretation on the full DAG to get reliable results.” Let me says that the last phrase can be rewritten as: so it's not enough to have a probabilistic model, you need structural causal equation/model (SCM). Pearl asked us to keep in mind a demarcation line between statistical and structural/causal concepts. The former alone can be never enough for proper causal inference, we need the latter too; here stays the root of most problems. In my opinion this clear distinction and his defense, represent the most important merit of Pearl.
Moreover Pearl suggests some tools such as: DAG, d-separation, do-operator, backdoor and front door criterion, among others. All of them are important, and express his theory, but all come from the demarcation line mentioned above, and all help us to work according with it. Put it differently, is not so tremendously relevant to argue pro or cons of one specific tool, it is rather about the necessity of the demarcation line. If the demarcation line disappears, all of Pearl's theory goes down, or, at best, adds just a bit of language to what we already have. However, this seems to me an unsustainable position. If some authors today still seriously argue so, please give me some reference about it.

I'm not yet expert enough to challenge the capability of all these tools, but they seem clear to me, and, until now, it seems to me that they work. I come from the econometric side and, about the tools therein, that I think the opposite. I can say that econometrics is: very widespread, very useful, very practical, very empirical, very challenging, very considered matters; and it has one of his most interesting topics in causality. In econometrics some causal issues can be fruitfully addressed with RCT tools. However, unfortunately, we can show that the econometrics literature, too often, addressed causal problems not properly. Shortly, this happened due to theoretical flaws. The dimensions of this problem emerge in their full width in:

Regression and Causation: A Critical Examination of Six Econometrics Textbooks - Chen and Pearl (2013) and

Trygve Haavelmo and the Emergence of causal calculus - Pearl; Econometric Theory (2015)

In these related discussions some point are addressed:

Under which assumptions a regression can be interpreted causally?

Difference Between Simultaneous Equation Model and Structural Equation Model

I don’t know if "equilibrium problems" invoked by Dimitriy V. Masterov cannot be addressed properly with Pearl SCMs, but from here:

Eight Myths About Causality and Structural Equation Models - Handbook of Causal Analysis for Social Research, Springer (2013)

it emerges that some frequently invoked limitations are false.

Finally, the argument of Matt seems to me not relevant, but not for "citations evidence" as argued by Neil G. In two words, Matt's point is

“Pearl's theory can be good for itself but not for the purpose of practice”.

This seems to me a logically wrong argument, definitely. Matter of fact is that Pearl presented a theory. So, it suffices to mention the motto “nothing can be more practical and useful than a good theory”. It is obvious that the examples in the book are as simple as possible, good didactic practice demands this. Making things more complicated is always possible and not difficult; on the other hand, proper simplifications are usually hard to make. The possibility to face simple problems or to rend them more simple seems to me strong evidence in favor of Pearl's Theory.

That said, the fact that no real issues are solved by Pearl's Theory (if it is true) is neither his responsibility not the responsibility of his theory. He himself complains that professors and researchers haven't spent time enough on his theory and tools (and on causal inference in general). This fact could be justified only in face of a clear theoretical flaw of Pearl's theory and clear superiority of another one. It is curious to see that probably the opposite is true; note that Pearl argued that RCT boil down to SCM. The problem that Matt underscores comes from professors' and researchers' responsibility.

I think that in the future Pearl's Theory will become common in econometrics too.

Answer 6

The synthesis behind causal reasoning is access to different aspects of reality/model using inferences from data. The paradoxical features in a model are avoided through formalisation in mathematical language. Informally, these counterintuitive features arise in observing a model unlike intervening through doing calculus.

The question on the openess of algorithms in the foundations of mathematics and logic is debated. One of the central point is that formalisations lead to harness infinite power to practice but turns these consistent procedures closed. In principle, while you do, you close the system unlike observing that has an active role in quantum mechanics unlike Pearl's causation hierarchy. Quantum contextuality asserts fundamental acausal relations between events, manifest as bugs! Bug could be a way into the depth of an iceberg! Technically, DAG are insufficient to reveal deeper structure of nature, where unrelated events could be related in a non-trivial way via topology.

Pearl's vision is significant in the sense that, he formally constructed causation theory and this integrity is elegant!