What can we say about models on observational data in the absence of instruments?

Question

I've had in the past a number of questions asked of me relating to published papers in a number of areas where regressions (and related models, such as panel models or GLMs) are used on observational data (i.e. data not produced by controlled experiment, in many cases - but not always - data observed over time) but where no attempt to introduce instrumental variables is made.

I've made a number of criticisms in response (such as describing issues with bias when important variables may be missing) but since other people here will no doubt be vastly more knowledgeable than me on this topic, I figured I'd ask:

1. What are the major issues/consequences of trying to come to conclusions about relationships (particularly, but not limited to causal conclusions) in such situations?

2. Can anything useful be done with studies that fit such models in the absence of instruments?

3. What are some good references (books or papers) on the issues with such modelling (preferably with clear nontechnical motivation of the consequences, since usually the people that ask have a variety of backgrounds, some without much statistics) that people might refer to in critiquing a paper? Discussion of precautions/problems with instruments would be useful too.

(Basic references on instrumental variables are here, though if you have any to add there, that would be helpful too.)

Pointers to good practical examples of finding and use of instruments would be a bonus but isn't central to this question.

[I'll likely point others to any good answers here as such questions come to me. I may add one or two examples as I get them.]

Answer 1

So the vast majority of my field (though not the part I work in most) is concerned with just this - the fitting of GLM-type models to observational data. For the most part, instrumental variables are a rarity, either due to a lack of familiarity with the technique or, as importantly, the lack of a good instrument. To address your questions in order:

1. The major issue is, of course, some sort of residual confounding by an unobserved variable that is associated with both the exposure and outcome of interest. The plain language version is that your answer might be wrong, but you don't necessarily know how or why. Decisions made on that information (like whether or not to use a particular treatment, whether X thing in the environment is dangerous, etc.) are decisions made using the wrong information.

2. I'd assert that the answer to this is yes because, for the most part, these studies are trying to get at something where there isn't necessarily a good instrument, or where randomization is impossible. So when it comes down to it, the alternative is "Just guess". These models are, if nothing else, a formalization of our thoughts and a solid attempt at getting close to the answer, and are easier to grapple with.

For example, you can ask how serious the bias would have to be in order to qualitatively change your answer (i.e. "Yes, X is bad for you..."), and assess whether or not you think it's reasonable there's an unknown factor of that strength lurking outside your data.

For example, the finding that HPV infection is extremely strongly associated with cervical cancer is an important finding, and the strength of an unmeasured factor that would bias that all the way to the null would have to be staggeringly strong.

Furthermore, it should be noted that an instrument doesn't fix this - they only work absent some unmeasured associations as well, and even randomized trials suffer from problems (differential dropout between treatment and controls, any behavior change post randomization, generalizability to the actual target population) that also get glossed over a bit.

3. Rothman, Greenland and Lash wrote the latest edition of Modern Epidemiology which is essentially a book devoted to trying to do these in the best way possible.

Answer 2

In contrast to the view from the epidemiologist's side shown by Fomite, instrumental variables are an essential toolkit in economics that is taught fairly early on. The reason for this is that there is a huge focus on trying to answer causal questions in economic research nowadays which goes to an extend where mere correlations are even regarded as uninteresting. The main limitation is that economics is a field were it is inherently difficult to do randomized experiments. If I want to know what is the effect of an early parental death on a child's long run educational outcomes most people would object to doing this via a randomized control trail - and rightly so. This handout from an MIT course outlines on page 3-5 what other issues there are with experiments.

To address each point in turn:

1. Depending on the question that is to be answered it is not just omitted variables that may invalidate analyses on observational data without the use of non-experimental methods. Selection problems, measurement error, reverse causality, or simultaneity may be equally important. The main issue is that the data analyst needs to be aware of the limitations of this setting. This refers mainly to the business case because in an academic scenario this would be uncovered quickly. Sometimes I see market analysts who want to estimate a price elasticity to inform a client (e.g. by how much does demand decrease if we increase prices by $x\%$), so they estimate a demand equation and completely forget or ignore the fact that demand and supply are determined simultaneously, and that one affects the other. So the consequences depend much more on the awareness of the researcher/data analyst with respect to the limitations of the data rather than the data itself, but the resulting consequences can range from something trivial to an extend where they negatively affect peoples' lifes.
2. Showing correlations can be useful sometimes, it just really depends on the question. When looking for a causal effect it is also sufficient if you have a natural experiment. The census data in Chile may be observational but if you want to know how the last earthquake affected educational attainment (where earthquakes are arguably exogenous) then also observational data is fine to answer a causal question.
   It is also possible to a certain degree to assess the endogeneity without instruments (see page 9 in the above handout, 'Estimating the extent of omitted variables bias'). For a binary non-experimental treatment $D_i$ you can compute the effect of this treatment, do the same for the unobservables and ask how large the shift in the unobservables must be in order to explain away the observed treatment effect. If the unobserved shift must be very large then we can be a little more trustful towards our findings. The reference for this is Altonji, Elder and Taber (2000).
3. Probably any applied economist would recommend Angrist and Pischke (2009) "Mostly Harmless Econometrics". Even though this book is mainly intended for graduate students and researchers it is possible to skip the maths parts of it and just get the intuition which is also nicely explained. They first introduce the idea of an experimental setting, then tend to OLS and its limitations with respect to endogeneity from omitted variables, simultaneity, selection, etc. and then extensively discuss instrumental variables with a good share of examples from the applied literature. They also discuss problems with instrumental variables such as weak instruments or using too many of them. Angrist and Krueger (2001) also provide a non-technical overview of instrumental variables and potential pitfalls, and they also have a table that summarizes several studies and their instruments.

Probably all of this was a great deal longer than a typical answer here should be but the question is very broad. I just would like to stress the point that instrumental variables (which are often hard to find) are not the only bullet in our pocket. There are other non-experimental methods to uncover causal effects from observational data such as difference-in-differences, regression discontinuity designs, matching, or fixed effects regression (if our confounders are time-invariant). All of these are discussed in Angrist and Pischke (2009) and in the handout linked to in the beginning.