How can someone improve a synthetic control?

Question

This is a broad question, I know, but I feel like it's ok to ask because I searched everywhere and I couldn't find the answer.

So I was studying about synthetic control (here: https://matheusfacure.github.io/python-causality-handbook/15-Synthetic-Control.html), and I thought about doing for an experiment I was running, but I ran into a problem: what can you do when your synthetic control is not so good? How can I improve it?

Let's use the book in the link as an example. There was a policy change in California and they evaluate the impact of the policy change by using California as the test group and by using all other states combined as a synthetic control group. The author did that by running a linear regression using the other states as the features and Califonia as the target. He then gets the coefficients from the regression and use it to build his synthetic group. But what if this coefficients are not so adherent? What if when you apply it to get the synthetic group, this group is not so similar as the place you are evaluating the change?

As already noted in the comments, there is one more step I didn't mention, the interpolation of the coefficients. Anyway, the problem is the same: let's suppose that you've tried the regression of all states and California, tried the interpolation, but the error is still big? I mean, suppose the regression+interpolation generated coefficients that didn't fit the data so well, suppose you applied the coefficients interpolated to actual data in order to see how well these coefficients could predict California data, but the error from the prediction these coefficients made is too big. Is there anything you could do to improve it? What should you do if you tried to build your synthetic control group but it didn't get so good.

Answer 1

You can sometimes improve fit by:

• Getting more pre-treatment data when possible, though that runs the risk of going so far back that the structural relationship is too different from the period in the study, like when CA was Mexico. Also, does not always work.
• Adding variables that are not just the lagged outcome in calculating the weights, like beer consumption and income per capita.
• Removing units from the potential donor pool that are dissimilar to the treated unit:
  • Only use untreated units that do not adopt interventions similar to the one under investigation during the period of the study.
  • Do not suffer large idiosyncratic/local shocks to the outcome of interest during the study period.
  • Have characteristics similar to the characteristics of the affected unit.
• Fit can sometimes be improved by using a transformation of the dependent variable (e.g., $\Delta Y_{it}$ or $\frac{\Delta Y_{it}}{Y_{it}}$, so you are matching the trend or growth rather than the levels. Normalizing by population can help if your treated unit is large relative to the untreated, like CA would be had they not done per capita.
• If the bias in fit is constant over time, you can simply subtract it from the effect. However, I have never seen that happen in the wild. -- If the poor fit is mostly in the early pre-treatment period, that period can be excluded if motivated well.
• If the poor fit is in the late pre-treatment period, that could indicate evidence of treatment anticipation. For example, if consumers knew the tax was coming soon, they could stock up and put the tobacco in a freezer. Then moving back the treatment start date to the tax increase announcement can help.

If the fit is still bad, abandon the project and move on with your life or try something else.

Answer 2

In addition to the other excellent answer (and yes, more data can really help, this could include different levels of aggregation, e.g. instead of state-level data, can you get county-level data?), there's also the option of shortening or weighting the time horizon for which you are matching the synthetic control. E.g. in the book example, they could have gone further back (assuming the data are available), but they clearly made a choice that e.g. matching the trajectory for data from before the second world war is irrelevant to the question at hand. You could also decide that on a sliding scale of importance matching the data 1987 is twice as important as matching the data for 1977 (which you can either achieve by weighting the loss functions or just scaling the values you are matching - e.g. scaling standardized values in 1977 by a factor of $\sqrt{0.5} \approx 0.707$). This would be arguing that a similar recent trajectory is just more important than what happened further in the past.

However, there's a situation where interpolated synthetic controls just cannot match the treated unit. That's when it's the most extreme of the available units. E.g. int he book example, if California had had by far the highest (or the lowest) per capita cigarette sales prior to the intervention, there would just no weighted average of the other states that can match its numbers. While there might be ways around it (e.g. if you go to the county level and there are some counties somewhere that can match the California counties), this would also have been a big red flag warning you that perhaps no other states really look like California and trying to compare what happens in California to what happens in these states is really problematic. I.e. it might be a warning that there might be no good answer. To quote John Tukey:"The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data."