2

I must evaluate the goodness of an instrument according to the measures of the final products. I have two instruments: one good and the other one whose goodness I have to evaluate. The measures are related to pressures.

What I thought is to sample same products with both the good instrument and the unknown one. The same product will be measured first with the unknown instrument and then with the reliable one. The instruments do not affect each other's measures; they are independent. What I will have is a paired sample.

Without any other information I assume to take about 35 measurements so that I can think of having normality (30 is not a magic number for data normality but the sample size should be defined by the power of the test adopted and I am not sure which test to adopt). Then I will take the difference between the two measurements obtained for each product (to purify the data from the variability of the individual products) in order to obtain a vector of differences which expresses the variability of the two instruments.

What I would like to know is whether the measures of the unknown instrument are significantly different from the measures of the good instrument, I thought about comparing the variance of the sample differences with an ideal population variance (to understand if there is such variability in the data that it can be attributed to different instruments or only to the case). Does this line of reasoning make sense? In this case, can I use F-Test? Which test do you recommend?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
filippo
  • 21
  • 1
  • Consider test that involve pairs and allows for correlation, the paired t test if normality is assumed or the Wilcoxon signed rank test for a nonparametric alternative, – Michael R. Chernick Oct 29 '17 at 20:34
  • Hi, thanks for your reply. The paired t test is usually used to determine whether the mean of the differences between two paired samples differs from 0, so is the idea of using variance not valid? – filippo Oct 29 '17 at 21:19
  • Power of the test will depend on variance but usually you don't know or have an estimate of it prior to testing. You can do one-sided tests that one mean is greater than the other by a fixed positive quantity. The test will have less power. – Michael R. Chernick Oct 29 '17 at 23:33

1 Answers1

2

You don't really want a hypothesis test in this case. What you are really after is a measure of agreement. What you ultimately want is for the two instruments to give the same value for the same product. In your case, you are taking the good instrument as a 'gold standard'. That is, you are defining the output from that instrument as correct, and comparing the unknown instrument to it.

If you thought that the quality of both instruments was relatively equal, and you wanted to assess agreement, the standard thing to do would be to use the methods described by Bland and Altman (pdf), especially by making a Bland-Altman plot. The gist of this is that you plot the means of the paired measurements on the x-axis and the differences between the measurements on the y-axis. In your case, by having a gold standard, you are saying that its measurement is correct. Thus, a simple adaptation is to plot the gold standard measurement alone on the x-axis, with the differences on the y-axis. From there, you want to see if the vertical spread is constant, straight, and level from low measured values of the gold standard to high measured values.

If you wanted, you could test the differences of the unknown from the gold standard with a one-sample t-test. That would be a test of whether the unknown instrument is biased. That would be a test, and people often assume the point of a study is to test something, but it isn't clear that it's the piece of information that should be most important to you. Under the assumption that the vertical spread of the differences was the same across the plot, presumably what you want to know is the standard deviation of the differences, so that you would know how reliable the unknown instrument is, relative to the gold standard. You could also form an $R^2$ type measure by dividing the variance of the gold standard measurements by the sum of the variances of the differences and the gold standard measurements. That would give you an estimate of the proportion of the total variability in the unknown instrument's output that is due to true variability among the products.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650