1

I have weights data of users collected over a period of time. My goal is to find incorrect weight readings. The definition of incorrect readings is purely based on logical reasons (or in other words based on common sense). For example, if today a user's reading is 60kg and the next day, the reading is 70kg, obviously one of them is incorrect as an increase of 10kg is highly unlikely in a span of single day.

To decide which one to choose (out of 60kg or 70kg), I am using moving median i.e. I calculate the median over past 10 days and checking whether the current reading falls between +-3 standard deviations, If not consider it an incorrect reading.

The issue arises when a user has recorded same reading for past 10 days, but on eleventh day, even if it increases by 1kg, which is likely. The function considers the 11th day reading as an incorrect reading.

My question or where I need help is: are there ways to deal with scenarios highlighted above with the example, if so I would really appreciate if you can share some insights on them.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
monte
  • 121
  • 3

1 Answers1

1

I'm not sure if using standard deviation is suitable in this problem since you have a small sample size (if I've understood you correctly: you are taking each 10 points to be a sample, finding the standard deviation and checking if the 11th point is beyond a certain threshold?).

Another method you could try is the modified Z-score method: $M_i = 0.6745(x_i - \tilde{x})/MAD$ where $MAD$ is the median absolute value. If the value $M_i$ exceeds $3.5$ then you can classify the point as an outlier. As I understand it, this should be more suitable for smaller datasets. See: https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm for a reference.

If you are interested, this thread discusses why using the standard deviation to remove outliers is not always a great idea: Detecting outliers using standard deviations.

Depending on the amount of data that you have, you may also want to look into time-series specific methods for outlier detection (https://neptune.ai/blog/anomaly-detection-in-time-series), or using clustering based techniques (https://www.knime.com/blog/four-techniques-for-outlier-detection).

namiyousef
  • 43
  • 5