4

I am interested in using a spatial research design. Imagine a line, like a time zone line. For example, in the United States, the line that makes between Eastern Standard Time and Central Standard Time runs North to South through the U.S. (and other places), more or less.

Suppose the United States implemented a policy where, with respect to county boundaries, all counties west of the 75th longitude were to get a tax deduction. No counties to the right of it would. I want to evaluate the impacts of the this tax deduction on crime.

For the sake of argument, can we please assume that the discontinuity is valid. I am trying to understand this idea for spatial discontinuities, in general?

I can imagine proceeding one of two ways. One way would a regression discontinuity. Exploiting the distance of say a county from this line. The other way would be a matching of counties that straddle this line on opposite sides. Note that a county might appear more than once if it borders more than one county on the opposite side of the time zone line.

The regression discontinuity includes a larger sample because it may include counties that are not exclusively along the border. The matching hones in on counties that touch each other, whereas the regression discontinuity, at best, can partition the time zone into smaller segmented lines where segments likely include multiple pairs of counties along the time zone.

What are the trade-offs of the research design? Which is preferred? Does one offer something that the other doesn't?

dimitriy
  • 31,081
  • 5
  • 63
  • 138
user1690130
  • 755
  • 2
  • 11
  • 23
  • What is your goal from this? What are you trying to show, in layman's terms? That is, are you trying to show that being in the central time zone vs. eastern time zone is related to some other property of counties? Or what? – Peter Flom Oct 15 '12 at 13:51
  • @PeterFlom Thank you Peter for this question---I should have clarified earlier. I just expanded the initial post with: "Suppose the goal is that an event happened in the Central Time Zone that did not happen in the Eastern Time Zone. My goal is to evaluate the impact of that event on say the county's average income, crime rate, and death rate." I hope that clarifies, but please let me know whether I should clarify further. – user1690130 Oct 15 '12 at 15:11
  • On the county level, you can do this without looking at time zone at all. If you have, say, data on the crime rate for each county, measured at many time points, then you can look at the effects of events using a multi-level model or possibly time series analysis. If you think being in a particular time zone affects crime rate, you can add that variable. Am I missing something? – Peter Flom Oct 15 '12 at 15:19
  • @PeterFlom Well, I'd like to exploit that there is an anticipated discontinuity, say at the time zone, per some event that happened. I could do these other things, but incorporating this discontinuity would make the results more compelling, I think. I just don't know the trade-offs between doing so as an RD or a matching pairs procedure. – user1690130 Oct 15 '12 at 16:02

4 Answers4

3

I would say matching is more appropriate (and easier), but the logic to each has some comparable aspects worth expounding upon.

Regression discontinuity designs are predicated on the fact that there is some observable relationship between some variable, $X$, and the outcome, $Y$. Then in RDD there is some other exogenous impact that occurs at some threshold of $X$. Note, implicit in the design is that cases are comparable on each side of the threshold (that is, no other differences between the cases exist on each side of the threshold), and so any discontinuity in the effect of $X$ and $Y$ before and after that threshold can be considered the treatment effect.

One of the things that makes RDD in this circumstance difficult is that it is unclear what $X$ is in your circumstance (you could think of many in addition to the distance one you mentioned) and for the social science variables listed, it is unlikely $X$ has a clear/obvious/strong relationship to $Y$. Also I would be skeptical that cases on either side of the threshold are entirely comparable, and so one would want to include other socio-demographic indicators. This can be done, but makes such a quasi-experiment markedly less appealing.

Thus I would suggest matching or estimating propensity score models. You can certainly find a history of examples of matching across the border (see for instance Card & Kreuger, 1994). Also I have seen matching spatial units extended to propensity score models, for instance Ridgeway (2006) uses a flexible set of generalized boosted models to estimate propensity scores for post traffic stop outcomes (e.g. searches, arrests). Such flexible models are attractive because spatial trends can be hard to characterize with such social science data and may take many parameters to effectively model. Also such models are readily capable of including other sets of socio-demographic covariates one would be expected to include in such research designs (for at least the outcomes you mention).

Citations

Community
  • 1
Andy W
  • 15,245
  • 8
  • 69
  • 191
  • Thank you for your thorough response! It is not clear to me though, why the RDD is less preferable. As I mentioned, I am interested in the impact of a certain event, which only happened in the Central Time Zone. For example, imagine that schools in the Central Time Zone all begin one hour later than in the EST, and I want to examine the impacts of this one hours difference on say student test scores and crime. The running variable, X, say time schools open, should change discretely at the cut-off at the time zone boundary. – user1690130 Oct 15 '12 at 16:27
  • @JG, Ok, so to put a face on it lets say you have a RDD where the easting of whatever unit is placed on the X axis, and the Y axis has the crime rate. So, true, there is a theoretical discontinuity at whatever easting the time zone break occurs. What happens if easting has little to no relationship to crime? What if there are more differences to cases on either side of the line than just difference in X coordinates (say on the east side have on average more poverty)? – Andy W Oct 15 '12 at 16:46
  • RDD designs are much stronger when the X variable has a strong relationship to the outcome *and* small variations in X are plausibly exogenous of other factors. In your example it isn't obvious that either can be assumed. – Andy W Oct 15 '12 at 16:46
  • For the sake of argument, can we assume that it is a valid discontinuity? So that either the RDD or matching would satisfy the identification requirements. Or, would you prefer that we do some other hypothetical example? Suppose we have some earthquake that exactly occurred in only the CT Zone and not at all in the ET Zone and we want to infer the impact of that earthquake. – user1690130 Oct 15 '12 at 16:51
  • IMO I would say it is a reasonable and valid discontinuity if there weren't any differences (i.e. imbalances on any covariates of interest) on either side of the line. I can think of a few that might occur with actual time zones (time zones frequently are nudged to not split cities or states, which may cause imbalances). Even if that isn't your real use case though hopefully that example is clear. – Andy W Oct 15 '12 at 16:57
  • Thank you for taking all of this time to help! Extracting from the example for the sake of argument, it is still not clear to me whether you recommend RD and subdividing the border into equal length segments for fixed effects or creating county pairs of neighboring counties across the boundary and using pair fixed effects. It sounds like you recommend the "matching" but it is not clear why that would be preferable to RD. – user1690130 Oct 15 '12 at 17:03
  • For the sake of argument, we can also assume I am interested in something like the Card-Krueger example you pointed to. I could either choose counties that border NJ and PA, with counties appearing more than once if they border more than one county in the opposing state. Or, I can treat the NJ-PA border as an RD, extending the sample deeper into both states. – user1690130 Oct 15 '12 at 17:05
2

I think the main tradeoff is that RD has high internal validity, but low external, and the opposite is true for matching. Since we don't have a great deal of detail about what you are actually doing, you will have to decide which one is better in your case.

The regression discontinuity design comes closest to an ideal experiment, so it has high internal validity, but unfortunately low external validity. The parameter it estimates is a very localized sort of local average treatment effect since you're mainly considering units right around the discontinuity cutoff. If you have reasons to think there are heaps of heterogeneity in the effect, you're stuck up the proverbial creek with no paddle.

Propensity score matching/reweighting methods can only eliminate bias due to selection on observables, and will yield efficient estimates of many types of treatment effects (average treatment effect or the treatment effect on the treated), so they often have higher external validity compared to RD, but the internal validity that is often questionable since it is rare that selection depends only on observables. If there's selection on unobservables, matching can actually exacerbate bias.

Finally, you might also want to consider an RD/Difference-in-Differences hybrid since you may have panel data on your counties:

\begin{equation} \hat\beta=\frac{(y_{1}^{+}-y_{1}^{-})-(y_{0}^{+}-y_{0}^{-})}{x_{1}^{+}-x_{1}^{-}} \end{equation} where $y$ is the mean outcome, the signs denote position relative to the cutoff and the numbers denote before and after, and $x$ is the mean treatment (may be 1 in the case of the deterministic assignment or sharp RD design).

Update: I just remembered a paper by Keele and Titiunik about geographic RD.

dimitriy
  • 31,081
  • 5
  • 63
  • 138
  • Thank you for your thorough response and for pointing out that I should have clarified further. I added to the post: "Suppose that schools in the Central Time Zone, on average, begin one hour later than in the EST, and I want to examine the impacts of this one hours difference on say student test scores and crime. The running variable, X, say time schools open, should change discretely at the cut-off at the time zone boundary." With that said, which approach would you suggest? Or would you suggest both? – user1690130 Oct 15 '12 at 16:41
  • Let me also clarify that the matching would not be on observable characteristics per se, but simply on the whether the county is along the discontinuity. So, the sample would be smaller than RDD. I would think external validity would be even weaker. But is the internal validity stronger? I would be using pair fixed effects for each set of neighboring counties. Note, that means that some counties may appear more than once if they border more than one county on the other side of the time zone. – user1690130 Oct 15 '12 at 16:43
  • To be honest, I am not sure I understand your example. Kids in both groups would presumably have the same number of hours in school. I can see how you might compare counties right at the border, and make the argument that the kids in the western half would be less sleepy, so they would learn more, and hence there would be less crime. Is that what you have in mind? – dimitriy Oct 15 '12 at 17:43
  • That is a good point. Would this be a better example (as I added to the question): Suppose the United States implemented a policy where, with respect to county boundaries, all counties west of the 75th longitude were to get a tax deduction. No counties to the right of it would. I want to evaluate the impacts of the this tax deduction on crime. – user1690130 Oct 15 '12 at 17:52
  • Thank you for editing your response to include that paper! It looks like the matching does not receive much treatment. I see some discussion on page 27. Is that because the matching is uniformly not preferred? – user1690130 Oct 15 '12 at 18:01
  • 1
    One of the assumptions of RD is that the other variables are smooth functions of the assignment variable conditional on treatment. I would worry that would not be the case with county and municipality data, so that would favor PS matching. In your example, there's also clearly no selection into treatment, so that would favor matching. – dimitriy Oct 15 '12 at 18:04
  • So, when is spatial RD appropriate? – user1690130 Oct 15 '12 at 18:10
  • They have another paper on what, at first glance, looks like a hybrid approach: polmeth.wustl.edu/media/Paper/GeoMatchVersion02.pdf. I have not read these two papers carefully, so I don't have a good answer to your second question. – dimitriy Oct 15 '12 at 18:17
0

The matching procedure you described can be thought of as a special case of regression discontinuity where you set the estimation bandwidth to zero.

Speaking somewhat informally to develop intuition - the problem of causal inference is that there may be some unobserved variables that are related to both your treatment variable and your outcome. In a regression discontinuity framework, the key assumption needed to establish causality is that none of these unobserved variables change discontinuously at the treatment threshold. The treatment effect is then the difference between the limit of Y as it approaches the threshold from the left, and the limit approaching from the right.

Calculating these limits in the real world requires us to make some choices in modelling how Y changes with respect to the running variable. In your case the running variable could be Euclidian distance to the time zone boundary. It's unlikely that this distance causally effects your outcome variable, however it could very well be correlated with some unobservables. In this case it makes sense to think about this in a geographic regression discontinuity framework. It is perfectly reasonable to include other covariates in the regression. In this sense, geographic RD can control for the same observables as a propensity score matching type technique, but it will also control for unobservables that change with distance to the border. Propensity score matching requires the somewhat stronger assumption that there are no unobservable confounders in order to establish causality.

The question then becomes how far from the border should we include observations. There is a tradeoff choosing this 'bandwidth' between bias and variance, and optimal methods for choosing the bandwidth have been developed (see Imbens and Kalyanaram 2011). If you restrict observations to the border counties, then you are forcing the bandwidth to be zero.

mdg
  • 1
-2

I may be mistaken so please feel free to correct me if this is not the correct answer. However, I think the correct answer is that both approaches would be acceptable but rather that the assumptions for RD to hold are more plausible, making it preferable.

Consider a research design in which both hold. Basically, RD only requires that the unobservable characteristics are continuous. The matching effectively requires that there are no selection on unobservables and matching by proximity would be adequate to take care of all unobeservables. While the matching probably does account for some unobservables--unobservables that RD would have to address--that RD only requires them to be continuous is probably more plausible. At the same time, RD estimates are more efficient.

Any reason that matching would not hold would also cause RD to not hold. Likewise, if spatial RD is plausible then so must spatial matching. So, when spatial RD is possible, it is preferable to spatial matching.

user1690130
  • 755
  • 2
  • 11
  • 23
  • "RD only requires that the unobservable characteristics are continuous" - This is not correct, reread the comments Dimitriy made and the long chat session we had. Also "Any reason that RD would not hold would also cause matching to not hold" - this is not true either. – Andy W Oct 26 '12 at 19:24
  • On the second point, you are correct. I meant the other way around. That if spatial RD holds then spatial matching must also hold and if spatial matching does not then spatial RD must not. On the first point, I mean that if both spatial RD and spatial matching are plausible designs, the assumption on the unobservables is that they are continuous for RD. – user1690130 Oct 26 '12 at 19:28
  • I guess I'm still not following then. It strikes me like when spatial RD is plausible (and it is not always), that would be strictly preferable for the reasons I provided given that argument that I provided above. I cannot see why the above argument is wrong. – user1690130 Oct 26 '12 at 19:34
  • 1
    RD requires other assumptions. First, individuals cannot manipulate the assignment variable. So there's no moving across the border for the better tax rate. More importantly, the other variables need to be smooth functions of the assignment variable conditional on treatment, i.e., the only reason the outcome variable should jump at the cutoff is due to the discontinuity in the level of treatment. That is not likely if the unit of observation is a county. It maybe reasonable if the units are people. – dimitriy Oct 26 '12 at 20:01
  • I take your point to mean that spatial RD is inherently not a good research design. However, wouldn't these problems also be a limitation to matching? They would presume that there are sources of selection that matching would not fully address? – user1690130 Oct 26 '12 at 20:11
  • Not quite. I think different hypothetical experiments *and* how you plan to use the estimated treatment effect render RD more (or less suitable) as an estimation strategy. Well-done RD is very convincing. It's easy to verify that some of crucial assumptions hold and graphs are also convincing. With matching, the CIA if often hard to hang your hat on, but to quote Jeff Smith, selection on observed variables can be true enough in particular, well-motivated contexts. On the other side, RD has low external validity and matching can potentially give you a richer set of treatment effects, ATT & ATE. – dimitriy Oct 26 '12 at 21:36
  • I agree with those statements for both RD and matching, in general. However, in this case of a spatial discontinuity, it is a little different, isn't it? What would be a setting in which a spatial RD makes sense but a spatial matching would make no sense, where the matching is done based on nearest neighboring units (counties, districts, whatever the unit) around the discontinuity? I am not referring to matching on observable characteristics per se, but strictly on nearest neighbors around the discontinuity. Thank you for all of your feedback! – user1690130 Oct 26 '12 at 22:12
  • This might be an example. Let's call the running variable S and assume we have a sharp RD design, which seems reasonable given your thought experiment. Now Pr[D=1|S] is either 0 or 1, whereas for matching you need 0 – dimitriy Oct 26 '12 at 23:30
  • I might not be following your example, but as I said earlier, by matching, I mean pairs of counties that touch each other, and not necessarily pairs of counties that overlap in otherwise measurable characteristics. For example, matching Washington DC to Prince Georges County, Maryland, strictly because they border each other. So, in that case, in ever pair, one county in the pair would be Pr[D=1|S] = 0 and the other would be 1. It would not be continuous per the usual cases of matching. This it the type of matching in the spirit of the Card-Krueger example listed earlier. – user1690130 Oct 27 '12 at 00:05