1

I have a question regarding outlier detection.

The dataset consists of monthly data for each location. So, for example, the first location "USA" will have values of [8,1,2,1,0] from Jan to May, "Japan" has [2,3,1,20,1], and so on, and I will have to examine whether the future month's data is an outlier when we obtain the data. So, we examine each month.

There seems to be many statistical methods like z-score method, median absolute deviation method, median rule, Tukey's box plot, and so on, which provides a way to detect outliers.

I don't want any machine learning algorithms like one-class SVM because the dataset is too small to train and I don't want to train each time the monthly data comes in.

My question is, there seems to be many detection methods out there including the ones I mentioned above, so how would you go about choosing which method to use?

Also, if you have general recommendations/advice about outlier detection, please let me know as well.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Jenny Bunge
  • 11
  • 1
  • Could you explain what an "outlier" might mean for your study and what action you intend to take upon identifying one? – whuber Jul 17 '15 at 13:21
  • ' and I will have to examine whether the future month's data is an outlier when we obtain the data. ' Can you explain this part better? I have a hard time understanding. Are you trying to predict whether an unseen data point will be an outlier? – user603 Oct 05 '17 at 18:10
  • Why not make the question more specific by giving an example? – user603 Oct 05 '17 at 18:18

1 Answers1

2

I think that i have helped procedures to do precisely what you want. Given data prior to the most recent value ...."what is the probability that the "new value" is different form what it was supposed to be ? Can you tell me the probability that a single data point (e.g. the latest reading) came from the distribution represented by all the previous data points? As Roger Bacon once said and I have often paraphrased : To do science is to search for repeated patterns. To detect anomalies is to identify values that do not follow repeated patterns. For whoever knows the ways of Nature will more easily notice her deviations and, on the other hand, whoever knows her deviations will more accurately describe her ways. One learns the rules by observing when the current rules fail.

The whole idea is to automatically develop an auto-projective model and then test whether or not the last observation Was consistent with the past. This can be done by developing an ARIMA model and the using Intervention Detection to asses the reasonableness of the most recent value.

IrishStat
  • 27,906
  • 5
  • 29
  • 55
  • 1
    How well would ARIMA be expected to work on a dataset that is "too small to train" on any machine-learning algorithm? – whuber Jul 17 '15 at 13:22
  • It would all depend on the signal to noise ratio from a tentative model. The greater the ratio the less the needed sample size required to identify a possible model. Consider the series 1,2,3,4,5,6,15 , or a series of the form 1,9,1,9,1,9,1,9,1,9,5 . The eye can immediately identify a possible model and subsequently flag the anomaly at the last period. A model based upon the n-1 values will suggest an anomaly at the last point. – IrishStat Jul 17 '15 at 13:40