5

Here I am not talking about medical research, where there is placebo effect need to be controlled with the blind design. In social science studies, we almost never do blind experiment. In this case, if we have data from all the subjects, do we really need to have a randomly selected control group to be compared to?

Say, if I have a teaching strategy to improve 5th grade students' math achievement and I want to test its effect. Student math achievement is measured by statewise standard assessments at every end of school year and all data will be extracted from district database directly. Which of the following is better:

  1. randomly select 20 schools from 200 schools in a district, and then randomly assign 10 schools to treatment group (use the new teaching strategy) and 10 schools to control group (do business as usual). Compare the control and treatment.

  2. randomly select 10 treatment schools from the same district. Compare the treatment with the rest of the district (control on previous achievement and demographics)?

  3. randomly select 10 treatment schools from the same district. Compare the treatment with 10 randomly selected schools (post hoc) from the same district (control on previous achievement and demographics)?

Is method 1 a better design and why?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Feng Jiang
  • 553
  • 1
  • 5
  • 11
  • 2
    Would you not still expect a placebo effect in this situation (that telling teachers to use a new teaching strategy would have a placebo effect on the outcomes)? – James Stanley Sep 10 '13 at 23:25
  • To amplify the comment by @James: None of those designs will work as intended, because they do not rule out alternative explanations like "the effects you think you see are due only to the fact of paying obvious attention to the treatment schools, not that the treatment works." Arguably, the fallacy of overlooking this possibility is at the heart of many failed social and political experiments (as well as underpinning a great deal of pseudoscience). – whuber Feb 17 '20 at 14:43

1 Answers1

2

If your control group was just going to be "business as usual" then by all means just use your out of sample as the control. From your description you're really just asking if you should use equal N's in control and experimental conditions. When considering the options you gave, the better defined control (large N) is the better solution.

However, quite often control is not business as usual. In that case you definitely need an assigned control group. Furthermore, control might be a matched control where the specific schools in the control group are matched as close as possible to the test group. When that happens your suggestion is not necessarily the best plan but that would depend on a number of factors.

John
  • 21,167
  • 9
  • 48
  • 84