1

I did some experiment in which tests are taken twice, pretest and posttest. I found there might be ceiling effect because the average of posttest is close to maximum test score possible. If I assume an IRT model: as the ability is getting higher (above difficulty level of the problems), the expected score distribution is skewed and never goes over the maximum score possible. So I think there might be some way to utilize the ceiling effect and skewedness of score distribution when comparing two averages of the test score.

But I wonder if there's any research done already about this subject, which could be called as "comparing two groups' averages assuming IRT model considering ceiling effect"...I am thinking of simulation research possibly with some MCMC.... Any idea or advice would be welcome also!

russellpierce
  • 17,079
  • 16
  • 67
  • 98
KH Kim
  • 1,121
  • 2
  • 10
  • 25
  • 1
    I don't know what models or methods psychometricians use or what they call them, but this is a common problem in survival analysis and also in economic data where things like income are often "topcoded" (there's a category like "makes over $180,000/year"). You want censored regression. Look, for example, at my answer to this question: http://stats.stackexchange.com/questions/83047/modelling-when-the-dependent-variable-has-a-cut-off/83059#83059 – Bill May 09 '14 at 15:06

1 Answers1

1

If you have an a priori response model you could attempt to simulate the range of plausible increases in underlying ability that could produce a distribution of effects like the ones you observed. It probably would be easiest to do a grid search through the space and then minimize a measure like Kolmogorov-Smirnov. I don't think that approach would be particularly standard though... but it might be close enough if you only have a practical need. However, no matter what you do you are going to be extrapolating beyond the observed values and having to lean on some uncertain assumptions. For example, you have no way to legitimately distinguish between an intervention that had a distribution of effects across students that was skewed and one that legitimately resulted from a ceiling effect.

russellpierce
  • 17,079
  • 16
  • 67
  • 98