3

This is my scenario:

I'm testing two length measuring devices. I'm measuring a model that has notches at different lengths in order to collect 15 different measurements. I know the "real" value for each distance in order to calculate 15 "errors" for each device. The same 15 measurements are repeated ten times for each device.

  • Device A - 15 measurements x 10 times
  • Device B - 15 measurements x 10 times

In order to have a general idea about which one is better I thought that a t-test would be ok (tell me if not): I put all the errors of Device A together and compare them with B.

If I want to compare A vs B of each one of the 15 measurements would it be ok to do a one way ANOVA? I'm asking it because I have only two groups.

Ferdi
  • 4,882
  • 7
  • 42
  • 62
Fed
  • 31
  • 4
  • 1
    Do the real values vary? (i.e. are they always measuring 15cm, or is it sometimes 10cm, sometimes 20cm, etc.) If that's the case then an alternative approach may be to calculate correlation coefficients for each device-real pairing, and look to see which has the larger coefficient. – Ian_Fin Sep 15 '16 at 10:01
  • I'm not sure I understood correctly. I have 15 "known" distances, eg. 13 mm, 14, 18, 18,6, etc... And I want to know which one is closer to the real distances – Fed Sep 19 '16 at 12:33

2 Answers2

2

If you just want to compare the differences between the two groups than a hypothesis test like a t-test or a Wilcoxon test is the most convenient way. There are some differences between statistical tests regarding small sample properties and how they deal with different variances. For reasons of simplicity I propose a simple t-test (welche two sample t-test).

Let´s have a look a two vectors. The first vector is called "a".

2 4 3 5 6 4 2 7 8 4

The second vector is called "b".

6 3 4 2 6 8 8 6 8 4

So you can use the following R command for testing. 

t.test(a,b)

    Welch Two Sample t-test

data:  a and b
t = -1.0674, df = 17.897, p-value = 0.3
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.9691637  0.9691637
sample estimates:
mean of x mean of y 
      4.5       5.5

The null hypothesis is that both samples have the same mean. The alternative hypothesis is that there are significant differences between the values of the two vectors.

One-way ANOVA however is applicable if you want to compare means of three or more samples. As you have only two samples you should not use a one-way ANOVA.

Ferdi
  • 4,882
  • 7
  • 42
  • 62
  • 1
    There are two issues with this approach. Firstly, depending on how the errors are summed the mean could likely be zero for both groups despite the devices varying wildly in their accuracy. Secondly, this assumes that both devices measure on the same scale. If the scales are different then two similarly (in)accurate devices could have different mean errors. – Ian_Fin Sep 15 '16 at 12:39
  • Thank you very much for your comment. Regarding the first issue: Of course one should have two compute the sum of absolute errors or the sum of squared errors. Regarding the second issue it would be presumably sufficient to transform one of the two vectors by dividing them or by transforming them using z-values, inverse hyperbolic sine or logarithmic transformation. – Ferdi Sep 15 '16 at 12:46
  • @Ferdi Thanks a lot For the answers. I applied the t-test for the "overall" comparison between the two machines. Nevertheless, what if I would like to perform statistics for each measure? – Fed Sep 19 '16 at 12:37
  • @Ferdi Thanks a lot For the answers. I applied the t-test for the "overall" comparison between the two machines. Nevertheless, what if I would like to perform statistics for each measure? here is a diagram of the measurements made [link] (https://s22.postimg.org/wuecmndch/frecce_Misuraz_001.jpg) Here is a sample of the 15 measurements made with Device A, repeated ten times. There is a similar table for Device B [link](https://s21.postimg.org/mwiad6cmf/Schermata_2016_09_19_alle_14_38_43.png) – Fed Sep 19 '16 at 12:42
0

Following extensive discussion in the comments with the OP, this approach is likely inappropriate in this specific case, but I'll keep it here as it may be of some use in the more general case

With your data you have three different measurements:

First, you have the "reference" measurement, i.e. the thing you are interested in measuring. For simplicity's sake, let us assume that this is known without error.

Second, you have the measurement taken from Device A. This is a measurement of the reference object which has some error.

Third, you have the measurement taken from Device B. Again, this is a measurement of the reference object which has some error (which may be more or less than the error with Device A).

The error associated with both measurement devices ensures that there will be variance in both sets of measurements. Furthermore, as you have a range of reference values (i.e., you didn't just measure the same thing multiple times) you'll have some variance in the reference measurement.

You could calculate a correlation coefficient between the reference measurement and the measurement from each device. The closer the coefficient is to 1 the more the variance in your measurements can be accounted for by the variance in the reference measurement, and therefore the less error there is (error is the variance that you can't account for by knowing the length of the object being measured).

For a specific sample, the device with the largest correlation coefficient (i.e., closest to 1), will be the less errorful device.

An example

As an illustration, I'll set up data for two measurement devices. One which is more errorful than the other

reference <- 6:20
measureA <- reference + rnorm(15, 0, .5)
measureB <- reference + rnorm(15, 0, 3)

And now, lets compare the measurements for each device with the reference measurements

enter image description here

It should hopefully be clear here that there is more error associated with device B.

Now, we can calculate correlation coefficients for each device compared to the reference

> cor(reference, measureA)
[1] 0.9954188
> cor(reference, measureB)
[1] 0.6487995

Note that the device with more error has a smaller correlation coefficient than the one with less error.

Of course, you may want to know whether the difference between correlation coefficients is statistically significant. This question may give you some help in that direction, although with only 15 observations the differences in reliability between the two devices may need to be large before you get a significant $p$-value.

Note:

For this approach, it won't matter whether the two devices are measuring on the same scale as the correlation coefficient is standardised.

Community
  • 1
Ian_Fin
  • 1,129
  • 8
  • 18
  • As a reference measure I have only one value. If I run correlation with SPSS duplicating ten times the reference measure, I get an error because one set of data (reference measure) is constant. If I place all the 15x10 measurements in one column, I can see the overall correlation but not each one of them. Am I missing something? – Fed Sep 19 '16 at 16:24
  • In your earlier comment you said that you had 15 known distances, which varied. The reference measures are these known distances. Am I misunderstanding something? As I understand it, you essentially have 15 distances which you've measured with each of your measuring devices – Ian_Fin Sep 19 '16 at 16:33
  • Thank you @Ian_Fin for the patience "15 known distances, which varied" --> right. Different segments with known distance (because i measured it with a reference machine). In the experiment, segment #1 to #15 were measured ten times each with both machines. I would like to be able to test significance between device A and B for each one of the segments – Fed Sep 21 '16 at 09:06
  • @Fed So you have 15 different segments of known, and varying, distances, and for each measurement device you have 15 measurements (one for each segment)? I don't understand where the duplication comes in, unless you measure each segment multiple times with the same device – Ian_Fin Sep 21 '16 at 09:10
  • Yes I do: I repeated the scan of the whole object (that has 15 measurements points within) ten times for each device. For each one of the 15 segments, I have 1 real value, 10 values for device A and 10 values for device B – Fed Sep 22 '16 at 06:49
  • [link](https://docs.google.com/spreadsheets/d/1D5mOvK0mGShvtSr5oMHVk-L8U76Vxy4GFqiEsJubHT4/pubhtml?gid=0&single=true) Here are my data: column B to P are the fifteen measurements. Rows represent repeated measures. @Ian_Fin – Fed Sep 22 '16 at 08:02
  • @Fed so, for each measurement device, you would compare the measurement with the actual reference value, and calculate the correlation coefficient. Strictly speaking, this approach might be considered sub-optimal because there is a dependency within your measurements that isn't accounted for (i.e. that two measurements of the same reference are more similar than two measurements of two different references), but I don't know the best way to deal with this. – Ian_Fin Sep 22 '16 at 08:22
  • @Fed If you're looking for a "best approach" then you'll need to edit your original question to make the details of our discussion explicit (e.g., that you have multiple measurements). That might get people to re-examine your question again. – Ian_Fin Sep 22 '16 at 08:23
  • If I'm doing right, performing a correlation between Device A and Reference value for, e.g. column B would have eight values for A and a constant value for Reference, which returns an error... Can you tell me how would you perform correlation with my data? Sorry but I don't have much experience @Ian_Fin – Fed Sep 22 '16 at 08:47
  • @Fed you would turn your 8x15 matrix into a single column of 120, and then you'd compare each measurement with the reference that you measured. So you'd have two columns: Measurements and reference values. – Ian_Fin Sep 22 '16 at 08:51
  • [link](https://docs.google.com/spreadsheets/d/1D5mOvK0mGShvtSr5oMHVk-L8U76Vxy4GFqiEsJubHT4/pubhtml?gid=1895542332&single=true) [link]![correlation](http://s21.postimg.org/7pxb4vjwn/correlation_8x120.png) – Fed Sep 22 '16 at 09:24
  • @Fed Yeah, that's how you'd arrange the data. You'd then do something similar for the other device and compare the correlation coefficients (although it may be that the number of decimal points that SPSS reports correlation coefficients in may be problematic for what you're wanting to do). I'd still suggest editing the question though. As it stands you have to read through all the comments to understand what you're asking. If you want the best response all of this information should be in the question itself. – Ian_Fin Sep 22 '16 at 09:47
  • But in this way I only get a single significance value, instead of 15 separate analyses. Anyway I'll try to change the title, following your advice :) – Fed Sep 22 '16 at 09:54
  • @Fed To be blunt, I would suggest putting some work into improving the clarity of the body of the question as well. Arguably, a better way to phrase the question is to ask what is the best way to compare the reliability of your two devices – Ian_Fin Sep 22 '16 at 10:21