0

So hypothetical situation is as follows: Let's say I make a neural net that can determine the length of an object in an image in centimeters. I also have an actual length for that object.

My goal: Determine if the length the neural net predicted is significantly different from the actual object length. Or, perhaps more formally, null hypothesis = that the predicted lengths (from neural net) are not significantly different from actual lengths.

My initial guess for figuring out how to best make a statistical test for this is to get the difference between the predicted measurement and the actual measurement for each sample, and see those differences are significantly different from 0 via a one-sample t-test; however, I am unsure if this is actually appropriate or not. But then again, I am unsure if this is appropriate since a 1 sample t test seems to talk about population means and I don’t have that here.

I’ve also thought about doing a paired t test but I am also unsure if that’s appropriate here since I think paired t tests tend to be done after some treatment/procedure being done and I don’t think that’s similar to my scenario. Does anyone have any advice?

To give some example points in this situation:

Item 1: predicted length=5.1cm; actual length=5.0cm.
Item 2: predicted length=50.5cm; actual length=50.9cm.
Item 3: predicted length=0.5cm; actual length=1.0cm.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
user2615936
  • 13
  • 3
  • Maybe this is more about measuring agreement? – kjetil b halvorsen Jan 20 '21 at 03:38
  • I guess so. Since I am dealing with continuous measures, what would be the appropriate significance test? Would it still be my proposed method or something else? – user2615936 Jan 20 '21 at 18:44
  • But what is your hypothesis? Exact agreement is improbable, so the null of exact agreement you can just reject (without need of some formal test.) So it seems more useful to measure *the degree of agreement*. Correlation is not a measure of agreement because with a constant non-zero bias you can still have correlation 1. See https://stats.stackexchange.com/questions/310635/which-test-do-you-suggest-to-use-to-catch-the-differences-in-paired-sample – kjetil b halvorsen Jan 22 '21 at 17:33
  • Isn’t this just the loss function in a supervised learning problem? – Dave Jan 22 '21 at 17:57
  • @Dave not necessarily. I'm specifically looking for a significance test. I'm personally more than happy with just a loss function, but my PI is looking for a p-value. – user2615936 Jan 24 '21 at 05:33
  • But you’re looking at the p-value for differences in loss, aren’t you? – Dave Jan 24 '21 at 07:41
  • I guess so, but I am unsure what statistical method I should use to get the p-values (i.e. should I be using a t-statistic from a paired t-test, a one-sample t test, etc) – user2615936 Jan 24 '21 at 19:20

0 Answers0