1

It is always emphasized that a significance test cannot prove a null hypothesis, only disprove it. Do you know any authoritative source (like a text book in statistics) that explains it and could be cited in a scientific paper?

The problem is that in validation studies (e.g. when validating that a new device is accurate), significance tests have been commonly used for PROVING null hypotheses. The logic is to prove that X=Y (measurements and real values do not differ) by fitting linear regression, calculating the coefficient of determination (R^2) and performing t-tests with the null hypotheses that the slope is 1 and the intercept is 0. Then if both tests give insignificant results (with p>0.05) and R^2 is sufficiently large (e.g. >=0.90 or >=0.80), it is concluded that the measurements were accurate. The data size is often very small, only 10-15 data points.

whamalai
  • 126
  • 5
  • 1
    Why would a statement in a textbook be authoritative? Many of them have errors. The only thing that has any real authority is a clear, convincing argument. – Glen_b Sep 11 '17 at 10:48
  • The paired t-test is for testing $\mu_X = \mu_Y$ and not correct if you want to test $X=Y$ ($X=Y$ implies $\mu_X = \mu_Y$ but the other way around is not neccesarily true). This is what might be the reason for the comment of the reviewer. See also: https://stats.stackexchange.com/questions/109191/inconsistent-x-y-relationship-from-paired-t-test-and-linear-regression ... – Sextus Empiricus Sep 11 '17 at 15:26
  • ... So far regarding regression vs paired t-test. The ideas about high p-value indicating acceptance of $H_0$ is a long story. P-value do not provide much information to "*accept*" the $H_0$, but the $R^2>0.9$ condition does say something about the applicability of $Y=X$ (even if p<0.05) and may be a suitable condition for making decisions. It may be interresting if you give more information on the references/problem to help direct the discussion. For instance, how bad would it be if $Y=1.01X$ and because of that p values are below <0.05 (which happens if you test with large $n$) yet R^2>0.9? – Sextus Empiricus Sep 11 '17 at 15:27
  • Consider your null hypothesis $H_0:$ Slope is equal to 1. You do the appropriate calculations and find that the p-value is 0.843. Now consider an alternate null hypothesis, namely $H_0:$ Slope is equal to 1.000000001. We expect the p-value to be similarly large. The data are consistent with both hypotheses (and many others), so it is improper to "accept" either one of them as absolute truth. Rather we say the data are consistent with your $H_0$. If we lived in a universe in which $H_0$ is true, we would not be very surprised to observe your data. – klumbard Sep 11 '17 at 17:32
  • Many of the tested data sets differed considerably from linearity. E.g. in one data set, the (least squares) fitted linear equation was y=0.01x+36.4, R^2=0.00 (and precision=0.02, sensitivity 0.02). The best fit was y=0.93x+5.74, R^2=1.00 (precision 0.97, sensitivity 0.91). We calculated also many other indices. – whamalai Sep 12 '17 at 11:05

3 Answers3

1

Doing this detour of a linear regression looks unnecessary to me if you can get paired data. You could also just subtract the paired measures of both instruments and have a single column of data to do a t-test on.

If you want to understand the reasons for not being able to prove the null hypothesis, read up on power analysis. It can broadly be framed as a problem of conflict of interest. As a researcher, you have a research hypothesis in mind that you want to prove. This is $H_a$. In order to arrive there, you need to produce evidence that rejects $H_0$. (Per set theory, $H_a$ and $H_0$ combined need to cover all possible outcomes.) This evidence needs to be strong enough for your p-value to express that the data would have been unlikely to occur under the assumption that $H_0$ is true.

If you want to prove $H_0$, you will be rewarded for sloppy work. The less evidence you collect, the more likely you are not able to reject $H_0$, which was your goal in the first place. In the worst case you actively ignore existing evidence against $H_0$. In the extreme, a researcher that never collects any data at all, will be able to prove all his precious $H_0$ research hypotheses and get published without doing any actual work.

Due to the mathematical nature of such tests, you can also not simply switch $H_0$ with $H_a$. For your purpose of showing a quantity does not differ from a given value, equivalence testing has been developed. It has the disadvantage that you need to define a parameter $\delta$ which represents the maximum acceptable distance to the value of interest the measures can have to still be considered equivalent. With two one sided tests (and a Bonferroni correction), you can show that your sample mean is neither smaller than the value of interest - $\delta$ nor bigger than it + $\delta$, thus equivalent.

That $\delta$ needs to be defined before looking at the data. Otherwise you have a similar conflict of interest again, the statistician would just select the smallest available $\delta$ for which he can prove equivalence. Women are as heavy as men if you take a $\delta$ of +-15kg around the weight of the average man... In this case, the scale is intuitive and the scam obvious. In most research scenarios, you could fool a reader by manipulating $\delta$. Equivalence testing is acceptable if you have principled reasons to justify $\delta$, otherwise it does violence to the epistemic principle of hypothesis testing.

David Ernst
  • 2,799
  • 8
  • 14
  • Yes, that's something I myself understand, but I would need an easy to read reference to an authoritative source to cite. The problem is that a reviewer (from biosciences) complains: 'Why didn't you do the linear regression analysis like authors A, B and C?' I know the problem has been widely discussed in medical science, but I am afraid papers from another field of applied science wouldn't suffice. – whamalai Sep 11 '17 at 08:49
  • Could you link me one of those papers where they do a regression for this. I'm actually interested in their reasons for the detour. It will help me argue better with it and add that to my response. What about a purely methodological paper that shouldn't be limited to any field? Do your reviewers like those? – David Ernst Sep 11 '17 at 13:09
  • Here is one of them: http://www.journalofdairyscience.org/article/S0022-0302(16)30472-6/references It deals a slightly different problem, validating a sampling protocol (like the other papers). However, it doesn't use any stochastic simulation as I would do. In our own study, we were validating a classifier and we could use confusion matrices and related performance measures. For comparison, we calculated also numerical measurements per hour and checked what results the traditional methods, like error indices and linear regression method would yield. – whamalai Sep 12 '17 at 10:56
  • An equivalence test sounds a good idea. With this kind of devices, the maximal acceptable error is usually 10%. It is also reasonable to allow larger errors if they are rare (e.g. at most 5% of measurements). How should we proceed here if we want to compare accuracies in different data sets? Fix 10% error margin and calculate the probability that it is exceeded? – whamalai Sep 12 '17 at 11:30
  • Since your delta is defined in percent instead of absolute terms you need to transform your data first. Take one of the instruments as reference r1 the other as r2. Transform your data as (r1-r2)/r1 and test if this number is equal to 1 with a delta of 0.1 (or 0.05 if 10% meant 5% in each direction). – David Ernst Sep 12 '17 at 11:49
  • Another possible transformation would be to (max(r1,r2)-min(r1,r2))/max(r1,r2) and test whether this number is bigger than 0.9. This is a more literal interpretation of "10% error is allowed". Always at most 10% of the larger measure of the two. Here you wouldn't even need an equivalence test, just a single one sided t-test to see if the quantity is larger than 0.9. But this approach is less conventional as it defines the difference sometimes as r1-r2, sometimes as r2-r1. It means that you really don't have a reference measure to compare two, just two measures. – David Ernst Sep 12 '17 at 11:52
  • In any case, look into the Bland-Altman plots as well, it can show you information about how the errors vary over the spectrum of measurement, a t-test cannot do that. – David Ernst Sep 12 '17 at 11:54
  • 10% error should be from the real value. It's just conventional. – whamalai Sep 12 '17 at 19:59
  • One problem in testing: The observations are not independent (consecutive hours) and there are quite many outliers. I don't know about normality, yet, but each numerical data set (from a single device and test situation) consists of only 24 observations (1 per hour, 24 h). With confusion matrices and their performance measures it doesn't matter, but I have understood these properties would ruin virtually any significance test? – whamalai Sep 12 '17 at 20:23
  • Those are indeed problems I am not an expert in – David Ernst Sep 12 '17 at 20:56
1

In the specific context of validating a new device, regression can make a lot of sense. As a reviewer I would want to see analysis of a calibration curve comparing the results from the new device against known values (or values determined by a highly accurate and precise method). This would be important for documenting the limits over which the output from the new device bears a linear relation to true values and for demonstrating how the magnitudes of measurement errors are related to those values.

If the comparison can only be done between two measurement methods that both have appreciable errors, analysis of what's known in the medical literature as a Bland-Altman plot (Tukey mean-difference plot) could be more appropriate. In this context of comparing devices, restricting analysis to t-tests of paired differences between measurements would throw out very important information about how differences might change over the range of measurement.

In any case, however, you never "prove the null hypothesis" even if that's the way some uninformed people might casually put it. You simply document that the output from the new device is unlikely to differ enough from the true value (or from the results provided by the older device) to make an important difference. Calibration via the method noted in the question effectively makes some hidden assumptions about how big a discrepancy makes an important difference. Proper analysis would provide confidence intervals for the differences between the outputs of the devices. For more background on this type of problem, you might want to look into non-inferiority testing.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • You can not 'prove' the null hypothesis, but you can determine that it is 'more probable', not?. Confusing here is that the question title mentions the situation whether a significance test can support the null, but in the text additional statistics are used such as $R^2$, and indeed one could use confidence intervals or other methods. (Should it be clarified that the core problem is about 'significance' $\equiv$ 'proof $H_0$', and that other analyses get more close to a casual putting of 'proof the null hypothesis'?) – Sextus Empiricus Sep 12 '17 at 08:04
  • @MartijnWeterings I think these general issues are fairly well covered in the thread that I linked at the end of my answer. – EdM Sep 12 '17 at 13:51
  • EdM, you are right. I was nitpicking. The link suits the purpose sufficiently. – Sextus Empiricus Sep 12 '17 at 14:11
0

Here is one source (from medical science) that gives a counter-example that the linear regression method doesn't work: Abhaya Indrayan: Clinical Agreement in Quantitative Measurements Limits of Disagreement and the Intraclass Correlation, Chapter 2 in Methods of Clinical Epidemiology, Springer 2013.

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwjOofXPvqDWAhVoCZoKHWvIDMMQFggvMAA&url=http%3A%2F%2Fwww.springer.com%2Fcda%2Fcontent%2Fdocument%2Fcda_downloaddocument%2F9783642371301-c2.pdf%3FSGWID%3D0-0-45-1402889-p174984284&usg=AFQjCNHtxad27T-bNsQhZDC5jbiEkoTjJQ

In the example, one can fit a perfect linear regression line y=x (with correlation r=0.945) but still there is no agreement in any of the subjects. The errors were considered substantial in this context.

whamalai
  • 126
  • 5