6

Let's say I got a Gaussian Process model $M$ based on some training data. Now I get a stream of sample data of a certain batch size coming in.

The GP does not model a time series, but it's trying to regress the value at certain locations $x$, that will be visited mutliple times.

I know that at some point there will be an abrupt change in the distribution the data batches are generated from. (At least on certain locations $x$)

Now I'm looking for a statistically sound way of detecting this change, that is I want to find the point, when the GP doesn't model the data well enough anymore.

I thought I would base this on detecting "big" changes in the likelihood function $L(\theta | x)$. However I'm not sure how to interpret "big" here, as the values of the likelihood function only have a meaning in comparison, but on their own.

Note: I'm asking this, because some people said my original question was too abstract. I didn't want to change the whole question, though, as there already were some answers.

Community
  • 1
Tobias Domhan
  • 263
  • 1
  • 10

2 Answers2

1

What do you think would cause the change? Is it a change in the mean, the variance or something else? The type of change you expect should determine the parameter you should test for change. Is this a sudden change? If so this could be done using intervention analysis for time series such as is done with Box Jenkins models for statiomary Gaussian time series. The autobox software can handle this for you.

Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143
  • I forgot to mention that the GP is not modelling a time series, but it takes in (x, y) pairs with x being the location, that will be visited multiple times. The changes will be abrupt, but they will only be at some locations $x$, whereas at the rest of the state space the distribution stays the same. I hope that is answering your question well enough. I will look for the Box Jenkins models you mentioned, thanks for the pointer :) – Tobias Domhan Jun 22 '12 at 11:49
  • I was only thinking of time series and not spatial-temporal processes. – Michael R. Chernick Jun 22 '12 at 11:55
0

So following is the solution I came up with. Please correct me if I'm wrong :)

Assumptions

The model change is abrupt.

Idea

My idea was the following: We determine the goodness of the fit by means of a model comparison. So we create a very simple model for which we know how to calculate the likelihood. As soon as this models becomes more likely than the original GP model we assume there was a model change.

We assume the data $D$ is normalized to have a zero mean.

null model likelihood

As a simplification of the GP we choose a normal distribution with zero mean and a large variance at each test location.

$\log p(D|M_0) = \sum\limits_{d \in D}\log p(d_i|M_0)$ with $d_i = (x_i, y_i)$

$\log p(d|M_0) = -\frac{1}{2}\log(2\pi)-\frac{1}{2}\log(\sigma^2)-\frac{1}{2\sigma^2}x_i^2$

GP model likelihood

For a GP with covariance matrix $K_y$ the likelihood is given by:

$\log p(D|M_{GP}) = -\frac{1}{2}\mathbf{y}^\top K_y^{-1} \mathbf{y} -\frac{1}{2}\log |K_y| -\frac{n}{2}\log{2\pi}$

Making a decision based on the Bayes factor

Assuming a uniform prior over the models, we calculate the log Bayes factor as follows:

$\log B_{01} = \log p(D|M_0) - \log p(D|M_{GP})$

According to this paper if $\log B_{01}$ is greater than 2 we assume there was a change in the underlying model.

Tobias Domhan
  • 263
  • 1
  • 10