3

Assume there is a predictor x (a video-recorded job simulation) that correlates r=.3 (Pearson r) with a criterion y (later job performance).

Assume a new grading process is used and it is noticed that there is a correlation of .25 between the order of grading and the grade, such that job candidates who are graded later are given somewhat higher grades than those who are graded earlier. (This correlation is statistically significant.)

Assuming the two effects are independent (i.e., the proficiency of the job candidates is unrelated to the order of grading effect), how might I go about predicting the degree to which the correlation of .3 is attenuated by the correlation of .25?

Joel W.
  • 3,096
  • 3
  • 31
  • 45
  • did you look into partial correlation issues? see wiki for example: https://en.wikipedia.org/wiki/Partial_correlation – Yuval Harpaz Jan 06 '20 at 15:18
  • @YuvalHarpaz . I have no criterion data. The correlation of .3 is from the published literature. The correlation of .25 is from the grades for actual candidates. Thus, no partial correlation is possible. – Joel W. Jan 06 '20 at 22:47
  • @JoelW. could you be more specific about the data you have available? I'm not sure what data the two situations (videos/performance, grading order/grade) have in common. – Drew N Jan 10 '20 at 17:34
  • @DrewN The video recorded job simulation (the predictor) is graded by experts, The correlation of .3 is the correlation between predictor and job performance (the criterion) is from published literature. I suspected some laxity in grading and noticed a correlation of .25 between order of grading job candidates who take the job simulation and the grades given those candidates. Does that help or is it still unclear? – Joel W. Jan 12 '20 at 00:12
  • I believe you are saying there is a chain of causes as follows: $$\text{order of job grading} \to \text {grades} \to \text{job performance}$$ with the first arrow carrying a correlation of 0.25, the second carrying a correlation 0.3. – Drew N Jan 12 '20 at 01:57
  • @DrewN I have 2 correlations, determined at different times with different samples. One correlation of .3 between grades and job performance. The second correlation of .25 between order of grading and grades. The correlation of .25 is due to faulty training of raters which resulted in a correlation between order of grading and grade. I presume the correlation of .3 was based on adequate rater training. The question is, how much does the order of grading effect attenuate the .3 correlation. Assume independence wherever possible. Your diagram shows a full model, but I lack data for that mod – Joel W. Jan 13 '20 at 13:18

2 Answers2

0

What if you look at the partial explained variance. For example run first the intial regression. And then run a regression adding the effect of grading order. I guess you could say something about the change in explained variance

Janosch
  • 530
  • 2
  • 10
  • I have no criterion data. The correlation of .3 is from the published literature. The correlation of .25 is from the grades for actual candidates. Thus, no partial correlation is possible. The question says to assume independence. – Joel W. Jan 06 '20 at 22:46
0

The implication seems to be that the grades are made into noisy predictors by the order of the grading, so what is the correlation between the "denoised" grades and the job prediction? But I don't quite know how one would model the way the grades receive that noise, and even if you do make such a model, I believe you need the actual data on job performance to evaluate the new correlation.

This is because one could imagine that the order of grading and the grades themselves are correlated in a way that makes the grades less accurate, but still not actually worse for the purpose of predicting performance. For instance imagine that the actual best workers were simultaneously the ones graded last, so that your correlation wouldn't at all be attenuated. This is unlikely of course, but because of this issue, I don't see how to untie the relationship without criterion data to test on.

Drew N
  • 490
  • 3
  • 10