1

I tested an open source people counting sensor vs actual observed people counts. I only tested on 1 location for 10 days and I divided the location into 8 different zones. I was expecting the error rate which is calculated using this formula ((Csensor - Cobserved)/Cobserved) to be uniform in each zone and be between (-1, 1) at least. However I noticed that the sensor significantly undercounts (sensor data mostly 0) in all zones except one where it significantly overcounts (sensor overcounts by as much as 50x). Here is a link to a copy of the raw data for more info: Data

Anyway is there a statistic test that I can use to prove that the sensor is broken (sensor data isn't related to observed data)?

kyljmn
  • 13
  • 2
  • 1
    Probably a paired t-test - assuming the data from each zone and each day is independent, which I think is reasonable here, will give you 80 paired counts as samples – Elenchus Sep 04 '20 at 08:32

1 Answers1

2

Welcome to Cross Validated!

The situation you have here is very common, and falls under the category of reliability testing. You have a gold standard for the counts of people and an estimate from a machine. This is the case in developing diagnostic tools in medicine as well as many other applications.

Here is a nice review of methods: Method agreement analysis: A review of correct methodology

That being said, your situation is more complicated than the models presented there.

  1. You have multiple measurements (days), not just a single attempt at agreement.
  2. You have sets of non-independent measurements. I'm assuming that the observed counts in the different zones are correlated.
  3. You have count data, which you could consider as ordinal or perhaps continuous as an approximation if you have to.

This review delves into these sorts of situations and explains why simplistic approaches (such as assuming the zones are independent) will not work.

These CV questions have further advice and links.
Internal reliability for an ordinal scale
Inter-rater reliability for ordinal or interval data

abalter
  • 770
  • 6
  • 18