2

Suppose I have a trend such as the following. Here is some code I wrote to generate fake data:

rm(list=ls(all=T))
set.seed(10)
fake.means = c(9,9,8,6,5,4,5,6,4,6,6,5,4,5,4,6,5)

df = data.frame(m=fake.means,sd=NA,N=NA)
for(i in seq(1,nrow(df))) {
  df$N[i] = sample(seq(10,50),1)
  df$sd[i] = sample(seq(0.1,2,by=0.1),1)
}
df$se = df$m/sqrt(df$N)

x = seq(1,17)

plot(x,fake.means,ylim=c(0,15))
lines(x,df$m,lwd=3)
lines(x,df$m+2*df$se,lwd=2)
lines(x,df$m-2*df$se,lwd=2)

polygon(c(x, rev(x)), c(df$m+2*df$se, rev(df$m-2*df$se)),col = rgb(0,255,0,255/2,maxColorValue = 255), border = NA)

It generates the following plot: enter image description here

Suppose I want to run some statistics to see if this trend is dependent on time (the x-axis). What would be the best thing to use and what would be the best way to do this in R? My initial thought was an ANOVA, but I'm not sure that's the best choice. I'd like to separately be able to run statistics on the first data points to demonstrate statistically that there is in an initial decline. Any suggestions? I appreciate your help in advance!

CodeGuy
  • 453
  • 3
  • 7
  • 15
  • My question to you is "How did you exactly form these 17 values" ? What underlying signal did you inject and expect to recover ? It appears to me that you took a constant and .682 of the previous value ( starting at point 2) and threw in two anomalies – IrishStat Dec 13 '16 at 18:25
  • the constant was 1.64394 . thus you used Y(t) = 1.643964 +.682* Y(t-1) for observations 2 through 17 plus two anomalies – IrishStat Dec 13 '16 at 18:33
  • I just randomly picked those 17 values as an example. – CodeGuy Dec 13 '16 at 18:45
  • not so randomly .... it would appear auto-correlation of lag 1 = .55 showing persistance (memory) . We are creatures of habit .... – IrishStat Dec 13 '16 at 18:47
  • Your objective " quantifying and describing the overall trend" is decidedly not met with the two trend models. – IrishStat Dec 13 '16 at 18:58
  • If you want to talk about the results please let me know . – IrishStat Dec 16 '16 at 21:35

1 Answers1

0

Your 17 data points suggests the need for a hybrid model incorporating not only memory (ARIMA) but an adjustment for 1 or more data points that do not follow the ARIMA regimen. Here is what might be a "useful model" enter image description here using AUTOBOX ( R version is available) , a piece of software which I have helped to develop that speaks to the challemges you raise. Since this data is relatively trivial in it's complexities there are some free remedies in R including auto.arima which might help you in this simple case. The problem with auto.arima is that it doesn't challenge the exceptional values but believes them and then gets confused as it attempts to form an ARIMA structure.

. The issue/objective is plain and clear ... determine the best composite between memory and deterministic structure. Here is another view of the model enter image description here .

THe Actual/Fit and Forecast graph is promising .enter image description here and the Actaul/Cleansed graph is informative enter image description here . The plot of the reenter image description heresiduals suggest sufficiency with 17 points . You can only do so much with 17 points and 13 degrees of freedom ! I suggest that you try different remedies ( R software solutions) on this data and then increasingly become more aggressive in simulating real-world data with level shifts , time trends , changing error variance , changing parameters , pulses , seasonal pulses et al and see how that works out for you. This can also be done while incorporating effects of X (eXogenous series).

The confidence limits on the forecast reflect re-sampling of the model errors possibly incorporating future anomalies .

Moenter image description heredel expressed as pure right-hand side equation

P.S. For pedagogical purposes, I specified a TREND MODEL (deterministic trend) and rejected any and all possible modifications and obtained the following simple/inadequate solution enter image description here . As another example of what not to do , I specified a sytochastic trend model ... first differences plus constant .. and obtained an equally poor soultion ..enter image description here

Following the request for a specific naive procedure . Assuming 1 and only 1 trend and no step shifts ....

Deterministic Trend ; Y(t)= B0+ B1*T + e(t) where T is the counting numbers (1,2,..T) and e(t) is the errror process and B0 and B1 are the least squares coefficients.

Stochastic Trend ; Y(t)= Y(t-1) + B2 + e(t) and e(t) is the errror process and B2 is the least squares coefficient.

See @RHARDY comments Are seasonal differencing and polynomial trends interchangeable?

add a comment | show 3 more comments

Community
  • 1
IrishStat
  • 27,906
  • 5
  • 29
  • 55
  • Hi there, thank you very much. I was mostly interested in quantifying and describing the overall trend as opposed to forecasting. Would you be able to show me your specific R code for those last two graphs? I'm interested in the step where you specified a trend model. – CodeGuy Dec 13 '16 at 17:10
  • In either case one needs to form a model representing the underlying signal i.e. quantifying and describing the overall trend . Detection of anomalies and/or forecasting are simply optional byproducts. – IrishStat Dec 13 '16 at 18:00
  • all models are wrong ... but this one seems useful ... whereas the two trend models are dis-functional. – IrishStat Dec 13 '16 at 18:55