2

I have a large number of time series which consist of pricing data of consumer goods. As expected the prices show trend and seasonality. However my main problem is to detect large level changes in the price series. In these data sets this has occurred mainly due to the price being recorded for the wrong packaging size. For example instead of the price for a can of beer being recorded, the price for a six-pack has been recorded.

For any given time series it is likely the case that most prices are correct, but a series of observations may be wrong. Most likely the errors are at the beginning of the series but not necessarily.

This is easy to pick up when the time series is plotted and can be modeled with a dummy variable.

However I would like to automatically detect the location and magnitude of the level change. How can I go about this ? Unfortunately my stats education ended at the undergrad level so I'm not sure where to begin ?

The trend and seasonality that exists in my time series is not my main concern. Do I need to worry about autocorrelation etc ? Or should I just be worried about level changes.

Although I have access to R, ultimately the algorithm may have to be implemented in Java.

I'm not sure if this is an appropriate place for the question, but I hope someone can help me !!

Antonio2100
  • 135
  • 7

1 Answers1

1

The Swiss Government http://www.autobox.com/cms/index.php/news/133-052306-used-in-swiss-foreign-trade-indices-users-guide-federal-customs-administration-fca-statistics-section-view uses commercial software that I have helped develop to cleanse Price data, much like what you describe. Detecting level shifts should not be done without detecting Pulses as they can confuse the automatic procedure. Detecting Pulses should not be done without taking into account any and all ARIMA structure i.e. auto-correlative structure that may be in the data (isn't it always ! ). Time Trends when present also provide a stumbling block to the detection of level shifts. The presence of Seasonal Spikes ( e.g. a June effect / a Friday effect etc ) can also cause the procedure to be confused. If the ARIMA process has transient parameters or if the error process is non-constant, these can also confuse the procedure. If the data is impacted by events/holidays etc then this can also play havoc with the automated procedure. I lay these things out to help you understand the scope of your project and what lies in front of you . Hope this helps !

IrishStat
  • 27,906
  • 5
  • 29
  • 55
  • This was helpful, so I marked it as the answer. It seems as if the arima.rob() procedure in S+FinMetrics provides robust fitting of arima models and is able to detect the types and locations of any outliers. Unfortunately I haven't been able to find an equivalent package in R. Regrettably I have no budget for S+ or autobox or similar software :-( . And lack the expertise to program this myself. – Antonio2100 Aug 11 '13 at 05:29
  • Well perhaps we could find n academic pricr for you. Please constact me at my emailaddres on ny cv – IrishStat Aug 11 '13 at 06:10