-2

I have a lot of time-range values, simple statistics chart, but some time can meet wrong values, for example:

[1000, 1010, 1025, 1005, 1, 1015]

or

[1000, 1010, 1025, 1005, 1030, 1, 2, 5]

Can you advise any logic or algorithm to determine the wrong values?

UPDATE: wrong values - values that will dramatically differ from previous and always will be lower. It should not differ more than twice, e.g. next value can not be twice lower than previous.

Roma Rush
  • 105
  • 2
  • 5
    Identifying wrong depends on identifying right. Nor is it obvious without more detail that extreme values are wrong. Hurricanes may be rare but they are real. So, this isn't well posed. – Nick Cox Oct 05 '17 at 11:42
  • 3
    Thank you for the update. However, it appears your second example does not conform to your criteria: neither the 2 nor the 5 are "dramatically different than [the] previous" value, nor are they lower than what precedes them. – whuber Oct 05 '17 at 13:49
  • 4
    Sorry, but "dramatically differ" is not a quantitative rule. A rule "twice lower than previous" (small problem of English wording there?) may rule out 1 following 1030 but it doesn't rule out 5 following 2 or 2 following 1. (== @whuber comment). – Nick Cox Oct 05 '17 at 13:50

2 Answers2

5

The answer will depend heavily on any background information you can give about the right values and how the wrong values appear. Just lines of numbers are never "right" or "wrong" but if you could say, that the "right" numbers are in the thousands and all data below 500 are wrong, than it is easy.

If the right data are usually close together and the wrong data far off (like in the first example you gave) then cluster analysis may help identify the rights from the wrongs.

UPDATE

Regarding your update: This translates straightforwardly to an algorithm. Write a computer program that finds values that are dramatically lower than the one before, where a reasonable value of 'dramatically' should be easy to find, if your usecase is similar to your example in whatever language you chose for the data evaluation. Unfortunately I don't understand "twice lower" but it appears as if it was easy to turn into an algorithm as well.

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
Bernhard
  • 7,419
  • 14
  • 36
5

You are probably looking for tools for outlier or anomaly detection in time series. Here's a Google search on it.

There are several classes of solutions for these problems, which I list a few at the end of the answer. The one I'm most familiar with are based on ARIMA models:

ARIMA

ARIMA stands for Autoregressive Integrated Moving Average and are models heavily used in Finance fields (among others) for modeling and forecasting time series.

To put it really simply, the AR part of ARIMA works on the assumption that the future values of the variable of interest depend on its own prior values while the MA part indicates that the regression error is related in time as well. The I part indicates that the data values have been replaced with the difference between their values and the previous values.

Here's an example of outlier detection using the R package tsoutliers:

Example of outlier detection using the R package tsoutliers

I find the website ARIMA models for time series forecasting a good overview of ARIMA models.

Other Time Series Anomaly Detection Algorithms are below:

  • STL decomposition
  • Classification and Regression Trees
  • Neural Networks
  • Exponential Smoothing
  • Neural Networks

Here are some links for further research.

Sources & References:

Community
  • 1
Ricardo
  • 165
  • 6