4

Imagine an experiment where you are trying to establish that cell phone users who receive a a local offer tend to spend more. How do you deal with those in the test group that cannot receive the treatment (say, there are no local offers when they're using their phone)?

For that matter, the same type of users might exist in the control group. How do you deal with subjects that cannot receive the treatment even if you wanted to give them treatment, but you only find this out after you've assigned them to control or test group?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
user765195
  • 2,075
  • 1
  • 20
  • 27

1 Answers1

4

If it isn't possible for the treatment to actually be applied, prima facie it makes less sense to include those data in the analysis. This suggests you should just throw them out as outliers—i.e., data that come from a data generating process other than what you are studying. However, I would argue that it's best to keep them. I would use one or both of the following approaches:

  1. Mark those who could not receive the treatment with an indicator variable, whether they were actually in the intervention group or not. Then fit a model with treatment.group, not.treatable and the interaction, treatment.group X not.treatable, all included. You can test the interaction, and the simple effect of being in the treatment group when it could possibly be applied. If you use reference level coding and make those who could not receive the treatment the reference level of the not.treatable factor, what the model will display as the 'main effect' of treatment.group will actually be the simple effect of interest.
  2. Ignore the fact that some people will not be able to get the treatment and simply test treatment.group (e.g., with a $t$-test). This is a test of the marginal association of being assigned to the treatment with the response. You could also just estimate the means of the treatment groups (again without taking whether people can get the treatment into account) and the standardized mean difference.

Number 2 may seem like an odd suggestion. In fact, you may not end up using it, but it's worth considering. There is a concept in biomedical sciences / epidemiology called "efficacy vs. effectiveness" (see also this CV thread: What is the difference between effectiveness and efficacy in determining the benefit of therapy 'A' on condition 'B'?). The idea is that an intervention has a possible pure effect which can be measured under ideal conditions, but which is likely to have a lesser effect in practice when conditions are not ideal, and that it would be worth knowing what both of those magnitudes are. In your case, 1 lets you test your treatment effect and tells you its 'true' (in some sense) magnitude, but 2 tells you how well to expect it to work in the future, since evidently you won't know in advance if the treatment can actually be applied when you try it.

Community
  • 1
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • Thank you very much for the clear and complete answer. It might seem surprising, but we are currently doing 2 and something about it seemed odd to me, hence my question here in SE. Incidentally, a friend mentioned the word "Counterfactual" in this context, but the only thing I could find about it was in the context of observational studies, rather than experiments. – user765195 Apr 05 '15 at 21:45
  • You're welcome, @user765195. – gung - Reinstate Monica Apr 05 '15 at 21:48
  • one more question: What is the difference between 1) and throwing out the untreatable cases? – user765195 Apr 07 '15 at 16:27
  • 1
    @user765195, 1 is more complete & honest, & 1 will have more power. – gung - Reinstate Monica Apr 07 '15 at 16:55