Is the Dunning–Kruger effect mostly caused by regression to the mean?

Question

In Bland's discussion of regression to the mean, there are several sections which detail examples of regression to the mean, which I understand to be an unavoidable consequence of not taking the mean of several measurements per subject. In the section entitled Comparison of two methods of measurement, Bland notes research which compared measured weights with self-reported weights. The conclusions of the research were that overweight people tended to underreport their weight whereas underweight people tended to overreport their weight. Bland says this is expected due to regression to the mean.

From my understanding, this comparison paradigm is the same as that used to establish the Dunning-Kruger effect which states that those who are unskilled believe they are more skilled than they really are, and those who are highly skilled believe they are less skilled than they really are.

It seems that, if Bland is correct, the Dunning-Kruger effect is more a statistical artefact and not necessarily reality. I've had a cursory read of the papers in support of the Dunning-Kruger effect and it seems that they do indeed rely on a single measurement for both "instruments" (i.e. self-reported vs measured).

Am I missing something or is there actually strong evidence of the Dunning-Kruger effect which is robust with regard to regression to the mean?

Answer 1

"Just an artefact" sounds like the effect could be entirely due to regression to the mean. That is a very strong statement. In fact, Kruger & Dunning already addressed this in section 4.1.3 in their original paper ("regression effect" here stands for "regression to the mean"):

Despite the inevitability of the regression effect, we believe that the overestimation we observed was more psychological than artifactual. For one, if regression alone were to blame for our results, then the magnitude of miscalibration among the bottom quartile would be comparable with that of the top quartile.

And that simply is not the case; their Figure 1 shows that the bottom performers overestimate their performance by more than the top performers underestimate theirs.

In addition, Kruger & Dunning explicitly ran additional studies (studies 3 and 4 in their original paper) to address this, and the results from their studies 3 and 4 are consistent with an actual underlying psychological effect.

Of course regression to the mean will have an effect in the basic Dunning-Kruger setup, and one could conceive a permutation test approach to estimate how much of the Dunning-Kruger setup is due to it. Nevertheless, my impression is that there is something more there.

Answer 2

Dunning-Kruger only showed that almost everyone, regardless of actual ability level (with the exception of those with very high ability), believes their own performance is above average. This isn't surprising at all, because people judge their own ability relative to the more typical person - because there are more typical people out there, then exceptionally rare people. DK did not show that low skilled individuals believe they are superior to highly ability individuals, which is a common and convenient misinterpretation of the study. It does suggest the low ability individuals believe they are 'relatively', a lot more able than they actually are.

Of course, one important consideration, is measures of 'ability' themselves. For example, in the test of 'logical ability', one should consider, in the light of recent studies on rationality and intuition, that an apparently high capacity for reasoning, on a test with strict time limits, does not translate into the probability of behaving rationally in real time. In other words, it has been established that how well a person performs on a test, does not translate directly into the probability that they will think rationally (and in this case, 'logically') at any given point of time. It has also been established by Chudhursky, 2012, that when time limits are relaxed on tests of fluid reasoning, that performance loses correlation with performance under strict time limits. In other words, some people perform a lot better when time limits are relaxed. All of this means, that artificially creating ability scales based on performance in a very narrow window of time, is a flawed practice.