2

I checked different questions on similar topics, but none were exactly the answer I wanted and I am confused.

I am working with big data, the data has a bursty nature with high frequency.

I considered features one by one with respect to time (equivalent to one time-series) and want to remove outliers in selected time series.

I am implementing this in java using weka. I read a lot about this problem but I did not find which exact method would work best for any time series. It would be also great, if any optimal outlier method could also smooth time series because at the end of day, I need to give multiple time series data to PCA for finding correlation. As you know, PCA is sensitive to outlier, noisy nature and missing values.

I know, I cannot find all three things in one algorithm but for me, outlier detection is tough part.
Please give your views on it.
Thanks

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
sash
  • 41
  • 4
  • I am no time series expert, but it seems to me that in a time series with bursts and outliers, much of the interest would be in the bursts and outliers. Why do you want to give this series to PCA? What is your research objective? – Peter Flom Jul 15 '13 at 23:12
  • Thank you for presenting this as a new question, @sash. I don't quite follow what you are asking, though. If you primarily want to know how to detect outliers in time-series data, that topic has been addressed several times on CV, most prominently here: [Simple algorithm for online outlier detection of a generic time-series](http://stats.stackexchange.com/questions/1142/). There are others as well; skim through the returns from [this search](http://stats.stackexchange.com/search?q=[time-series]+[outliers]+is%3Aquestion). – gung - Reinstate Monica Jul 16 '13 at 00:12
  • Hi Peter, Well I got this motivation from book "Data mining methods and models " by Daniel T.Lorse first chapter and many other research papers. – sash Jul 16 '13 at 00:56
  • PCA eventually express my target feature to many eigen vectors. Each eigen vector explain some correlation strength with target feature and each eigen vector has linear combination with remaining features (other than target feature). So we will extract feature correlation both on basis of particular eigen vector correlation strength with target feature and each eigen vector linear regression value with feature . So only first few eigen vectors gives ne correlated feature with target feature indirectly through eigen vectors. – sash Jul 16 '13 at 00:56
  • Hi Gung, thanks for your response. as it is written in R language and i have no experience in R. Do you think, i should try it to write in java. it is worthfull. and Second, should i take mean by stl in R to ARIMA model output? – sash Jul 16 '13 at 01:06
  • Why weka? It seems a bit clumsy to use Weka for this task. – Has QUIT--Anony-Mousse Jul 16 '13 at 08:20
  • because our customer have application written in java and I have to design this solution in java and will integrate with customer application . Is it make snede now ? Also weka have implmented all program in java. – sash Jul 17 '13 at 12:55

0 Answers0