4

I am writing a manuscript using an experimental design which predicts interactions between 1 continuous variable and multiple dichotomous variables, all predicting a continuous variable. As is traditional in experimental design, I have used ANCOVAs to analyze the data, but am concerned about the inability to specifically test the interaction between the covariate and the dichotomous variables.

Also, while I am expecting only a small amount of variance to be explained, there is only one significant finding among the multiple predictors and interactions. I suspect the non-significant findings may be due to the small amount of explained variance in the DV, which when spread across multiple predictors in the model is insufficient to distinguish between them.

A colleague has suggested that the best option to deal with both concerns is to do four 2 or 3 stage hierarchical regressions with dummy variables, i.e. each regression would comprise: stage 1: one dummy variable stage 2: add the continuous variable and the interaction between the two. After running these analyses, any significant predictors and interactions could be combined into a single hierarchical regression.

I am unable to find a precedent for this unusual procedure but interestingly, it reveals a number of significant results, albeit explaining a very small amount of variance.

Is this a reasonable procedure to follow, given the requirements of my variables?

Camille
  • 41
  • 1
  • 2
  • Hi Camille, welcome to the site. Your question looks interesting, but you're probably more likely to get better answers if your provide some specific details of what you're looking at. See http://meta.stats.stackexchange.com/questions/1479/how-to-ask-a-good-question-on-crossvalidated/ – naught101 Dec 10 '12 at 10:03
  • @naught101. Thanks for the direction. I thought it would be easier to answer if I wrote in abstract ideas rather than specific variables but here goes. I'm a statistician so just wrote – Camille Dec 10 '12 at 11:04

1 Answers1

6

ANCOVA and multiple regression are mathematically identical. In matrix algebra terms, both are $Y + XB + e$. If you don't have much variance explained in ANCOVA, you won't in regression.

The main problem I can see with your colleague's approach is that of inflating type I error by running multiple tests.

Statistical significance is, generally, of less importance than many people think it is. From a scientific/substantive point of view, if the effect sizes are small they are usually not interesting (although there are exceptions.... If you could reduce the number of, say, plane crashes by a small amount, airlines would probably be interested).

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • 1
    I have the same concern regarding my colleagues approach. I agree about statistical significance but unfortunately psychology is a field where the p value rules!! I should also say that I have no problem with reporting a lack of significant findings or with very small effect sizes as negligible results are informative in themselves regarding the lack of importance in the real world. – Camille Dec 10 '12 at 11:28
  • 1
    Oh, I know about psychology! I make my own small efforts to change things, but .... yes, p value is still (all too often) viewed as the most important (or even only important) thing. – Peter Flom Dec 10 '12 at 11:34
  • Clearly this is not ground-breaking stuff so small effect sizes are indicative that context doesn't mean much here. However I still have the question of how to test the interactions and whether the regressions are a better technique. – Camille Dec 10 '12 at 12:10
  • As I said, ANCOVA and regression are mathematically equivalent. Interactions may be easier to add in the regression model, but that would be because of notation and software implementation. – Peter Flom Dec 10 '12 at 13:06
  • 2
    +1. The issue is not ANCOVA v regression (they are the same) but how you build a model. The colleague's proposed approach sounds like a variant on stepwise regression, and will have all the associated problems as Peter Flom points out. See http://stats.stackexchange.com/questions/13686/what-are-modern-easily-used-alternatives-to-stepwise-regression . – Peter Ellis Dec 11 '12 at 00:14
  • @PeterEllis. Thanks. You are absolutely right that my colleague is suggesting a type of stepwise regression which is probably not appropriate here. As ANCOVA and Regression are mathematically the same, and all relationships are founded on a conceptual model, I think my issue really comes down to the way the results are presented to a potential reviewer who may be more used to seeing ANCOVAs. As I need the interactions to test the hypotheses, I am leaning towards stage 1: all single variables, stage 2: all interactions. Does anyone have a view on that? – Camille Dec 11 '12 at 10:51