12

Academic econometricians are often interested in determining causality. It seems like all the private-sector statistical/ data science jobs I hear about are only looking for predictive models.

Are there any jobs in the private-sector (or government jobs) that research causality?

Michael Webb
  • 1,936
  • 10
  • 21
  • 5
    Whenever we want to make an intervention, you bet we care. Think about all the A/B testing google does to make a simple design change. – Matthew Drury Aug 25 '16 at 22:06
  • 8
    Of course. Just about any legal case ultimately hinges on questions of causality. Almost any decent quality control scheme is concerned with causality. Engineers and scientists care a lot about it. – whuber Aug 25 '16 at 22:14
  • 6
    Another classic private sector question is, "Do my advertisements *cause* more sales?" – Matthew Gunn Aug 25 '16 at 22:43
  • 2
    @MatthewGunn: +1. In general: "Will this (costly) change make any difference?" Assuming a business needs to stay afloat (and possibly thrive) having some degree of understanding of the causal dynamics of its market-place is crucial. – usεr11852 Aug 25 '16 at 22:44
  • A big challenge that rises from this question, is that a lot of people think/say they care about cause, whilst actually they don't if you are looking at the situation with academic strictness. – Dennis Jaheruddin Aug 26 '16 at 09:04
  • 4
    I initially refrained from converting this thread to CW, believing it possible that an authoritative, data/fact-based answer could be offered. Since it's not turning out that way, for various reasons that many might find interesting and useful, I have converted it to CW. Thank you all for your contributions! – whuber Aug 26 '16 at 14:19

5 Answers5

15

I am an economist in tech that works on causal inference with observational or flawed experimental data. Most of the major tech firms will have folks like me around doing applied research on pricing, marketing, and product design. There are also public policy teams in some companies.

There are also lots of people that work on web experimentation. This is a much larger group.

Finally, there are also particular types of economics consulting, particularly anti-trust, where this the the main focus.

dimitriy
  • 31,081
  • 5
  • 63
  • 138
7

[The first five emotional replies censored.]

That is one of the strangest questions on the site, frankly. And shows how much disconnect there is between what your professors say and the real life -- that is, the life outside of the ivory tower. It's good that you are peeking out of it... but you (meaning, Ph.D. students in economics) definitely need to do this more often.

Yes, there are jobs outside academia where people (surprise, surprise) use causal inference methods. And (surprise, surprise) publish papers. My answers are U.S.-specific, but I am sure you can find similar organizations in other countries.

  • Example 1 (only because I am familiar with it internally at my job). I work in a subsidiary to a large contract research organizatoin, Abt Associates. It employs about 2,300 people in 50 countries, and most of them work on conducting or supporting evaluation research, and implementing interventions. One of the top 6 technical people (referred to as Senior Fellows), Jacob Klerman, is the editor of Evaluation Review, overseeing a board of editors of whom about 5/6 have academic affiliations. So that is a private sector example for you. (Check the company position ads to see specifically what kind of skills a company like that may be looking for -- I am not entirely sure everybody advertises at JOE as that's expensive; I can easily name another dozen in the U.S. who'd be happy to hire a craftsy econometrician.)
  • Example 2 (I have but a passing familiarity with that because I know people who started this project from other venues): What Works Clearinghouse at the U.S. federal Department of Education is a website devoted to meta-analysis of the published analysis of educational programs. WWC operates through a network of reviewers who are given specific instructions as to what is considered a study that has sufficient rigor to support causal claims, and what isn't. It turns out that most of that published research is absolute crap. As in, bullshit. No control group. No checking of the balance on the demographic covariates/baseline equivalence. Only about 3-5% of the studies (published in the peer-review literature, for goodness sake) "meet standards without reservations" -- meaning, they had some semblance of randomization, controlled attrition and cross-contamination of the experiment arms, and did the analysis in a more or less acceptable way down the line. (By Bayes theorem, when you hear somebody say, "But I saw it published that chewing gum increases math achievement", you can respond, "BS", and you'd be right 90+% of the time.) At any rate, this is a federal department project, so that's an example for you where a government agency reviews the proper use of causal inference tools. (Throw your name into the hat as a study reviewer, this will be a great educational experience for you. If I were teaching program evaluation, I would have made this a requirement for my students.) (For biostatisticians working with FDA, where you have to submit your analysis code before you collect any data, WWC standards are still very lax.)
StasK
  • 29,235
  • 2
  • 80
  • 165
  • I don't think economics professors say you don't use causal methods in practice (no one starts a talk with "here's some statistical methods no one will care about"), but rather the student is concerned that causal inference is just an ivory tower topic (such as log-concave density estimation: I assure you **no one** in industry does that, and for good reason). It's also not clear how example 2 shows people in industry using causal methods? – Cliff AB Aug 26 '16 at 20:50
  • @CliffAB The OP asked for industry and government examples, so #2 fits the bill. I also think StasK's point about scant knowledge of life outside ivory tower among economics PhD students, and to a lesser extent their professors, is pretty accurate, though there is lot of heterogeneity across fields and departments and even time. – dimitriy Aug 26 '16 at 22:31
  • @DimitriyV.Masterov: #2 seems like an example of **not** using proper causal tools. And I read (perhaps misread) StasK's answer as implying that professors are saying "no one outside academics uses causal methods". If a professor who specializes in causal methods said this, they are admitting failure; if you're creating applied statistical methods that no one outside of the academic world uses, that's not considered a good thing. Statistical theory is of course a different story. – Cliff AB Aug 26 '16 at 22:39
  • My reading (again, maybe misreading) of the OP's question is that the professors' are telling them "causal statistics is important!", and their response is "is it really important? Do people in industry actually use these methods?". But again, maybe I'm misreading. – Cliff AB Aug 26 '16 at 22:56
  • 1
    @CLIFF WWCH reviews academic research, separating the wheat from the chaff as far as causal inference, so it actually great example of an area where the standards are higher in government than in some parts of academia. – dimitriy Aug 26 '16 at 23:27
  • On the second claim, I have encountered two versions of this thesis: one is that these methods are so novel or hard that they have not percolated to industry, and people in industry misuse predictive models or experiments for counterfactuals. Both are arguably true in some form. You can see some of the first sort of soul searching in Rust's [review](https://www.aeaweb.org/articles?id=10.1257/jel.52.3.820) of Wolpin's book in the JEL, on p. 825. – dimitriy Aug 26 '16 at 23:28
  • @DimitriyV.Masterov: ah, okay. I was actually confused about WWCH: I read StasK's statement as saying that most of *WWCH*'s published work was crap, but now I understand that they are coming to conclusion that a lot of academic published work is crap (at least in certain fields). Sadly, I have to agree, but I'm happy to hear that something like WWCH exists. Thanks. – Cliff AB Aug 28 '16 at 20:32
  • @CliffAB, thanks for bringing this up, I made some small edits to incorporate your discussion with Dimitriy (who, as far as I know off-site, comes from economics background, and in many ways is more qualified to speak on the topic than I am.) Log-concavity is a fine property, though; my recollection is that some Monte Carlo sampling methods rely on it, but that is probably the closest it gets to the applied world. – StasK Sep 01 '16 at 13:46
6

In pharmaceutical statistics and a number of related fields the causal link between intervention and health outcome is the key question of interest when deciding whether an intervention should be used. There are a wide array of sub-fields such as randomized trials (clinical or pre-clinical), non-randomized or single arm trials, laboratory exerpiments, meta-analyses, drug safety surveillance based on spontaneous reporting of adverse events, epidemiology (including ideas like Mandelian randomization) and effectiveness research (e.g. using observational data such as insurance claims databases). Of course in the designed randomized experiments (such as randomized clinical trials) attributing causality is somewhat easier than in some of the other applications.

Björn
  • 21,227
  • 2
  • 26
  • 65
  • I suppose that a medicine development setting would be one of the few places where people care about cause of people getting better rather than whether they get better, because ultimately you need to 'be safe' in the whole population. -- So, definitely a nice answer, but as you mention, quite a special case. – Dennis Jaheruddin Aug 26 '16 at 09:11
4

I am a researcher at A Place for Mom, the nation's largest senior living referral service. We've designed a survey aimed at understanding how moving into an assisted living community influences quality of life. Causal inference is central to this research, and the methods of causal analysis (e.g., matching, modeling selection processes, estimating average treatment effects) are essential.

Brash Equilibrium
  • 3,565
  • 1
  • 25
  • 43
-3

In most private sector situations you will not care about causality

In practice, despite typical language use, people are much more often interested in well understood impact, rather than (well understood) causality.

From an academic point of view, it is very interesting to know:

If I do A, because of that the outcome will be B

But from a practical point of view, in nearly all situations the following is what people really want to know:

If I do A, the outcome will be B

Sure you may be interested in the impact of A, but whether it is truely the cause, or whether there is a hidden cause that just happens to create this correlation is usually not that interesting.

Note on limitations

You may think: ok, but if we don't know that A causes B, then it is very risky to work on that assumption.

This is true in a way, but again in practice you will just worry about: Will it work, or are there exceptions?

To illustrate this, you may note that this situation:

If I do A, in situation X, because of A the outcome will be B and because of X the outcome will deviate by delta

Is not much more helpfull than this situation (assuming you can quantify the impacts equally):

If I do A, in situation X, the outcome will be B and the outcome will deviate by delta

Simple example: Correlation to cause

  • A: Replenish engine oil
  • B: Reduced brake faillure
  • C: Car checkup

The logic: C always causes A and B

Resulting relation: If A goes up, B goes up but there is no causal relation between A and B.

My point: You can model the impact of A on B. A does not cause B, but the model will still be correct, and if you have information about A, you will have information about B.

The person interested in brake faillure with information about A will just care about knowing the relation of A to B, and only care whether the relation is correct, regardless of whether this relation is causal or not.

Community
  • 1
Dennis Jaheruddin
  • 436
  • 3
  • 20
  • 1
    I don't suppose I'm the only one who's confused by your distinction between "cause" & "impact". Your examples seem to clarify it a bit: unpicking the details of the causal chain might not be especially important. Buying more cost-per-click adverts will *cause* an increase in sales - never mind how - & that's what matters to your client. – Scortchi - Reinstate Monica Aug 26 '16 at 10:50
  • @Scortchi What I mean by cause: 'If A, then because of that B'. What I mean by impact (perhaps not the most typical definition, but this is not about language): 'If A then B'. -- Textbook example of a relevant difference: C causes A and B. Therefore A does not cause B, but I would say it would make sense to model the impact of A on B. -- After rereading my own comment, perhaps 'impact' could be replaced by 'actual relation with time lag'. – Dennis Jaheruddin Aug 26 '16 at 11:59
  • 5
    I don't think I agree with the advertising example - if I need to decide whether to buy more online ads, I care about whether doing so will result in more sales than not buying these ads (whether through more click-throughs or by other means=causality), which may be a different thing from whether companies that increased their online advertising in the last year had higher sales increases than companies that did not (correlation). Causality is not about the exact means by which the result occurs (would be nice to know,of course), but rather whether something occurs due to doing or not doing A. – Björn Aug 26 '16 at 12:22
  • 6
    In sales, you *should* care about causality. If you conduct an experiment in which action $A$ is followed by result $B$, *but that result is not causal,* then you you will be disappointed when--as is only natural and to be hoped--that when you repeat action $A$ in the future it will only inconsistently (if ever) be followed by $B$. In short, the distinction between "causes" and "impacts" used in this answer appears to be analogous to that between "statistically significant"--loosely interpreted as "worthy of belief and the basis for future action"--and "spurious." – whuber Aug 26 '16 at 14:17
  • I don't think I agree with this comment, though in my experience it is somewhat true causality is not sought after, though in many problems it should. In practice, I see this as nearly equivalent to the distinction between predictive models and explanatory models. – Thomas Speidel Aug 26 '16 at 14:32
  • @DennisJaheruddin: "If $A$, then because of that $B$" doesn't manifestly say any more than "$A$ causes $B$", & "if $A$ then $B$" can bear various interpretations. "$C$ causes $A$ and $B$. Therefore "$A$ does not cause $B$" isn't true taken at face value - $A$ could cause $B$ *as well as* $C$'s causing $B$, or $C$ could cause $B$ by means of $A$. I'm afraid I'm none the wiser. – Scortchi - Reinstate Monica Aug 26 '16 at 15:21
  • @Scortchi I have removed most examples and expanded the one that I found strongest. -- Perhaps A could even cause B, but the point is that this does not matter much, as long as you find the relation between A and B. – Dennis Jaheruddin Aug 29 '16 at 09:24
  • @whuber I am not sure whether I get your point. In statistics you typically care about significant relations between A and B. If they, for example, have a hidden common cause C that does not invalidate your statistics does it? -- Sure the relation changes if something happens to C, but in general relations will change if circumstances change. – Dennis Jaheruddin Aug 29 '16 at 09:34
  • So "impact" just means "association" or "correlation" or "relation"? I think it's liable to be misunderstood - in plain English when people talk about the impact of something they're talking about what it causes (or will cause or did cause) - & in any case a new word is hardly needed. Moreover the example doesn't seem to bear out the assertion that causal relations aren't usually of interest to the private sector - there must be a lot of people managing fleets of cars who'd care that car checkups reduce brake failures. – Scortchi - Reinstate Monica Aug 30 '16 at 08:44
  • @Scortchi You are right about the term. -- My claim is that those people don't care about the cause, as long as they know what the outcome of their decision will be. [If I use this maintenance schedule with option xxx, there will not be many faillures. Is that because of xxx, probably but who cares] – Dennis Jaheruddin Aug 30 '16 at 08:54
  • @DennisJaheruddin: They care whether adherence to a particular maintenance schedule *causes* fewer failures - that if they follow the schedule the outcome will be fewer failures & if they don't follow it the outcome will be more failures. If you're saying that perhaps they don't care about which specific features of the schedule cause this outcome, or about the chain of intermediate causes, you may be right; but that doesn't betoken a lack of concern about causality per se. – Scortchi - Reinstate Monica Aug 30 '16 at 11:23