5

I have a survey experiment in which the treated group receives a certain prompt that the control group does not.

I also have a bunch of demographic variables that I want to check for balance between the control and the treated group. A common technique that I've seen is a t-test comparing the two groups. However, if my sample size is large, doesn't this make the t-test more statistically significant regardless of the underlying balance between my two groups?

If that's true, is there a better balance test than the t-test?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Heisenberg
  • 4,239
  • 3
  • 23
  • 54

4 Answers4

3

The people in your study were randomly assigned to the groups. Therefore, unless there was a failure of randomization, we know by definition that the demographics are identical in the population.

It makes no sense to test if the demographics differ. The only two possibilities are that you get a non-significant result / make a correct decision, or that you get a significant result and thus make a type I error.

If there is reason to believe that the demographics are relevant to the response, you can still control for them in a multiple regression model. This would give you more power to detect differences due to your treatment.

If you want a reference, you might read:

  • Senn, S. (1994). Testing for baseline balance in randomized clinical trials. Statistics in Medicine, 13, pp. 1715-1726.
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • Isn't is possible to get highly imbalance groups by chance (e.g. all men in one, all women in the other?) This leads to the practice of re-randomization in practice (when researchers re-randomize until they got balance in covariates they deem important.) In my case though, I'm handed the data. – Heisenberg Mar 03 '16 at 22:37
  • 1
    Yes, it is possible there are all men in 1 group & all women in the other. So what? If you want to know the proportion male in your 2 groups, you just count how many men in each & divide by the totals. There is no uncertainty there. We conduct statistical tests to see if it is unreasonable to believe that the proportions are = in the population. B/c you randomized, we know a-priori that the populations are =, so all you can do is get a type I error. It is possible (not likely) to get all men & all women under the null; we know the null is true; so the p-value would show that this is unlikely. – gung - Reinstate Monica Mar 03 '16 at 23:04
  • 1
    I'm aware that researchers re-randomize, @Heisenberg. It is nonetheless foolish & unnecessary. If a variable is highly imbalanced, it would be due to chance alone. If that variable is also deemed important, you can & should control for it in a multiple regression model. But you do not need to re-randomize, & we know for a fact that the *populations* are =. – gung - Reinstate Monica Mar 03 '16 at 23:11
  • The problem in the male-female example is that male is perfectly correlated with the treatment. So we don't know that the difference between the treatment and the control group is due to the treatment or due to being male. We can't control in regression either since the two are perfectly correlated. This seems like a big problem to me but I could be missing something? – Heisenberg Mar 03 '16 at 23:14
  • @Heisenberg, if your treatment is *perfectly* correlated w/ a demographic variable, then the treatment effect could not be identified. That is correct. (Do you have a demographic variable that is perfectly correlated with the treatment?) Nonetheless, *conducting a hypothesis test* is nonsensical, since we know w/ absolute certainty that the null hypothesis is true / that the populations (not sample proportions) are =. – gung - Reinstate Monica Mar 03 '16 at 23:23
  • If a variable is perfectly correlated & you notice this before the treatment is actually applied, I suppose re-randomizing would make identification possible. Short of perfect correlation, statistical control will be fine. – gung - Reinstate Monica Mar 03 '16 at 23:23
  • 1
    In the case of ex post imbalanced covariates due to bad luck, if the re-randomization is free/cheap, does it make sense to re-randomize to get balanced groups? I understand hypothesis testing is meaningless. Just curious whether re-randomization would make analysis easier and less model dependent. – Paul Dec 28 '18 at 00:06
  • 1
    @Paul, what is the exact scenario here? Have the patients already had the treatment? If nothing has happened, you could presumably re-randomize & maintain the blinding w/o anyone being the wiser. You can also just include whichever covariate you believe is too imbalanced as a control variable in your model. You could ask a new question, & refer back to this 1, if you want more information. – gung - Reinstate Monica Dec 28 '18 at 03:03
2

You cannot test balance with a t-test, this is known as the ballance test fallancy - it is descibed in (Ho et. al, 2007) (http://gking.harvard.edu/files/abs/matchp-abs.shtml)

The main points is that: balance is a feature of the sample at hand, not some population - therefore p-values < 0.05 does not imply anything. Further, there is no point below which imbalance can be ignored, and even small differences between groups can translate into large bias. Therefore, you should always try to model differences in the (regression) model, even if the groups had almost perfect balance.

You question of sample size is speciacally adressed in the paper. You cannot use a statistic (t-test in this case) as an objective to minimize! Given a small sample a large difference could be insignificant, and given a large sample a small difference could be significant. You are therefore better off by measuring balance directly, by looking at the distribtuion of the variables (QQ-plots).

Repmat
  • 3,182
  • 1
  • 15
  • 32
1

Statistical significance is not the only way of interpreting a coefficient. While the large sample size makes it more likely that you will find a statistically significant difference between control and treatment groups you still need to look at the magnitude of the coefficient and determine if it is large relative to the problem at hand. Similarly, if you test for balance across many variables you will likely find some that are significantly different just by chance.

Chris Marciniak
  • 11
  • 1
  • It is more likely to find statistical significance with a large sample size when the null hypothesis is false, because the power increases. If the randomization was done properly, however, rejection of the null hypothesis would be a type I error, thus the probability to find statistical significance is 0.05 (or the chosen significance threshold) for each test you make. – Federico Tedeschi Feb 18 '20 at 09:48
0

Repmat is correct.

If you don't have demographic balance, your t-test is not testing just the treatment effect, it's testing the combined effects of (treatment effect + demo differences). "demo differences" = bias.

Sample size does nothing to alleviate this if the imbalance persists.

If you are stuck with imbalanced data with no principled way to legitimately clean / subset your data to acheive a balanced, representative sample, then model building is necessary to control for the imbalance.

meat wad
  • 1