2

I have been doing an analysis using a difference in difference setup. In my raw sample I use OLS and first difference (two time-periods) and I get the effect that I would expect. Namely that shocks in the stock market negatively affect the health of stock owners.

To mitigate confounding, I have tried using matching. Since the dataset is very large, I used only 10% of the original sample (else that matching would take forever). After matching I have 250K observations.

However, the effects are now insignificant. I figured maybe it was just a bad sample. So I ran a simulation, each time I draw a small sub sample (from the raw sample), matched, estimated and saved. This gave me a nice bell shaped curved around 0 - So I suppose the insignificance was not just a bad sample. The effect simply is not there.

The conclusion I drew based on the first analysis must be flawed in some way (picking up confounding factors, that just happen to fluctuate with shocks)… But I need some ideas on how to prove (or disprove) this.

Ideas?

Andy
  • 18,070
  • 20
  • 77
  • 100
Repmat
  • 3,182
  • 1
  • 15
  • 32
  • How did you do the matching, i.e. what did you match on? What exactly are the outcome and the treatment? – Andy May 17 '15 at 10:59
  • The outcome are health measures. The matching is done on treatment dummy. Indicating if someone is equity owner. The matching variables are; income, some demograpihcs, and social status. I used propensity score match. – Repmat May 17 '15 at 17:29

1 Answers1

1

There are many issues to this but I try to collect them with regards to the difference in differences set-up, the matching, and possible confounding factors.

You will remember from your other question why the parallel trends assumption is vital for difference in differences. With only one pre-treatment period it is impossible to check for parallel trends between the treated and control group. Another issue is that the treatment is not random. I don't know how you coded the stock market shock or how you define a "stock owner" but I would imagine that those are individuals who are conceptually very different from other types of investors, e.g. institutions (hedge funds, banks), small private investors and others. All of them are going to be affected in different ways by the treatment so it is hard to find a control group in the first place, but then they are also likely to not be very comparable. For example, institutional investors often possess a certain degree of inside information which makes them less vulnerable to some shocks. Of course all of this depends on what kind of shock you use, your control group, etc. but these are certainly things to keep in mind. The biggest problem though is the parallel trends assumption.

The matching is not a bad idea in the sense that you make your treatment and control units more comparable to each other. However, this only works if you observe all the other variables that make the treatment potentially endogenous. Often these so called "selection on observables" stories are not very credible because the selection usually happens on characteristics that are unobservable by the econometrician such as ability (more able individuals are likely to have a better diversified portfolio or make better use of available information and are thus less vulnerable to shocks). The point of matching would also be to reduce the sample by matching control units to the best fitting treated units and discard the less well fitting control individuals - if you have 50:50 treated vs control then there is nothing to match. Propensity score methods also often hinge on specifying the propensity score right, i.e. to have the correct functional form in your probit to estimate it. I'm also not convinced that your way of matching on subsamples is valid because i) you reduce the sample size so you also reduce the power of your statistical tests and ii) you are always potentially excluding better fitting control individuals from any given sample. However, I'm not entirely sure about this point but it seems not very natural to me to do this.

Regarding confounding factors you need not worry about the time-invariant ones as these will be differenced out in the difference in differences setting. A potential problem occurs if there are other policy or regime shifts at the same time the treatment occurs and that affect the treatment and control group in different ways. In your setting the bigger concern is about the parallel trends though because those are needed to get the right causal effect (if your treatment is in fact random, otherwise you will need an instrument for it).

Andy
  • 18,070
  • 20
  • 77
  • 100
  • 1
    Again in the mean time, from question to answer - I figured out some stuff. I asked for more data, and discorved that in the raw (unmathced) sample the common trends assunption was not "good" - significant effects where there should not be one. The mathced sample did not have this problem. I implemented matching, as per. Imai, Stuart, King & Ho. The article spefically says that reducing the sample size is a good thing, if there is large imbalance prior to the treatment. Otherwise thanks for the pointers. – Repmat May 18 '15 at 11:48