1

Problem: I have a single list of data that is updated with a new value every few seconds. I simply want to check whether the data points in that list are decreasing, increasing or staying steady over time. It would also be helpful to quantify that the strength of the trend.

So far, I have looked at the Pearson correlation and differences in values, but neither has helped (or I am not using them correctly). I also looked at other answers here, but I couldn't find anything that worked for me.

Question: Are there any other tests or calculations that I can do to determine the direction of the data? Would the moving average give any information?

This might be a relatively trivial question, but I would appreciate some help or pointers to the right resources.

Cuber
  • 111
  • 2

2 Answers2

1

The first stop would be a basic linear regression. Here is a simple example in R:

time <- 1:1000
x <- time+rnorm(1000)
m <- lm(x~time)
summary(m)

Call:
lm(formula = x ~ time)

Residuals:
    Min      1Q  Median      3Q     Max 
-3.6321 -0.7004  0.0009  0.6804  3.2288 

Coefficients:
              Estimate Std. Error  t value Pr(>|t|)    
(Intercept) -0.1388840  0.0639265   -2.173     0.03 *  
time         1.0001052  0.0001106 9039.197   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.01 on 998 degrees of freedom
Multiple R-squared:      1, Adjusted R-squared:      1 
F-statistic: 8.171e+07 on 1 and 998 DF,  p-value: < 2.2e-16

The coefficient estimate associated with time is your estimate of $\partial x/\partial time$, on average. If you wish to compute a trend on a subset of the data, simply subset the data that you feed to the regression.

Complexity increases from here, depending on more specific goals.

generic_user
  • 11,981
  • 8
  • 40
  • 63
0

The moving-average approach A.K.A. arima modelling can be used to compute the probability that the most recent values is statistically significantly different from expectations. Simply take the observed time series and add one new observation and then use intervention detection to assess the most recent value. Some software packages actually produce a file with the probability reported.

IrishStat
  • 27,906
  • 5
  • 29
  • 55
  • Thanks! Could you elaborate a bit on the intervention detection part? – Cuber Aug 08 '17 at 16:21
  • One would definitely not fit a time trend model and attempt to analyze the residual at the most recent point as the time trend model may be (NEARLY ALWAYS !) a bad specification.The Intervention detection scheme is to form an ARIMA model on the entire data set and then TRY a pulse X as an additional input variable at the last time period. If it is significant then that would suggest activity at the most recent point. The flaw here is that if there are unusual values OR level shifts OR time trends then one would need to also accommodate them in addition to the pulse variable at the last point. – IrishStat Aug 08 '17 at 19:23
  • if what I wrote is not enough you can either call me or set up a chat room and I will try and help further – IrishStat Aug 09 '17 at 13:09