How can I recognize dramatic changes in a set of observations?

Question

I'm trying to build a monitoring system that will automatically raise a warning when a dramatic change happens to some of the observed parameters.

My problem looks like this: We send out e-mails to a large number of recipients. For each pile of emails, we have a few parameters such as the number of e-mails that were sent as well as engagement counters such as opens and clicks, as well as bounces and unsubscriptions.

Typically, the number of emails sent per mailing would change slightly over time. The engagement ratios might stay more or less constant (accounting for variance, of course), or increase or decrease slowly over time.

Whenever there is a dramatic change in one of those metrics (such as bounce rates going up from 1% to 3%, while having been more or less constant before, or open rates decreasing from 30% to 20% while they were increasing slowly before), I want to be able to recognize this trend change.

I already employ static thresholds, but I want to identify outliers that might suggest a dramatic trend change. Which statistical methods are suited for solving this kind of problem?

Answer 1

Well, I think that you should use control charts, as suggested by AdamO. If you are not familiar with control charts, you could try a naive but simple approach: test if a new value in an influential one, if it "changes the trend". The Cook's distance may help you.

An example in R code:

> set.seed(1234)
> x <- 1:100                  # 100 observations to estimate the trend
> y <- rnorm(100, 10, 1)      # more or less constant values: mean = 10, sd = 1
> range(y)                    # min(y) = 7.65, max(y) = 12.55
[1]  7.654302 12.548991
> ### First scenario
> y_new <- 15                 # 15 is the new value, larger than max(y)
> y <- c(y[2:100], y_new)     # discard the first value, append the new one
> cd <- cooks.distance(lm(y ~ x))
> # Is the new value an influential one?
> cd[100] > 0.50              # a standard threshold
  100 
FALSE 
> ### Second scenario
> range(y)                    # y includes the previous y_new
[1]  7.654302 15.000000
> y_new <- 18                 # 18 is the new, and influential, value
> y <- c(y[2:100], y_new)  
> cd <- cooks.distance(lm(y ~ x))
> cd[100] > 0.50
 100 
TRUE 

I've used here 0.50, a standard threshold. If your record of past events is long enough, you can check if it is too low/high wrt your needs.

HTH, even if it really is a naive approach.

Answer 2

The procedure called INTERVENTION DETECTION speaks to the issue of empirically detecting a change in mean , a change in trend , a change in seasonal indicators and of course a 1 time change (pulse). Care has to be taken to account for any auto-regressive structure that may be present otherwise it can easily mask the differences. I recently posted on a problem/question similar to this. You should also note that to detect differences in means one MUST first account for any anomalies(1 time pulses) that may be present otherwise you might be unable to"see" the differences in the means. Changepoints in R speaks to Change Point Detection which is what I believe you are after. I would suggest that you make your data available to the list, so I and others might better help you. In my opinion dated procedures like Cook's Distance will be a far cry from what you need due to possible(probable) auto-regressive structure in the data and possible level shifts which can obfuscate the detection/