6

1- How can I check if a set of data can be assumed as IID data? I'm not so familiar with statistics, but I guess I should look at the first lag of autocorrelation for independent distribution. Have no idea about identical distribution condition!

2- It seems that I was not clear enough! I'm trying to detect outliers in a series of records (turbulent flow velocity in a river). I transform data into wavelet space and then I shrink the wavelets over a certain threshold. Since standard deviation is the worst option as an scale estimator, I looked for a new estimator. Rousseeuw and Croux developed new robust estimators for measuring dispersion in iid random variables, Sn and Qn. I don't know offhand if the high breakdown properties they enjoy carry over to the time-series case or not.

From the answer given by kwak, I can infer that wavelets do NOT follow independent distribution property. Since after shrinkage, location of non-zero elements indicates the spike location in the original time series. Am I true? (shuffling the indices results in losing location of spikes) If so, other scale estimators like median absolute deviation (MAD) are not valid in case of time series as we calculate the median.

How about identical distribution assumption requirements?

3- OK, let me ask my question in simple manner: I want to use robust scale estimators Sn and Qn for shrinking a series of wavelets. the wavelets are obtained from decomposing observations of a turbulent flow field velocity vectors collected at 1 Hz sampling rate. if the data can be assumed as iid e.g. Qn has breakpoint of 50% and efficiency of 82% (Gaussian distribution). My question is the high breakdown properties they enjoy carry over to the time-series case or not. Or how can i approve that the wavelets follow iid characteristics.

K-1
  • 237
  • 3
  • 8
  • IMHO lag auto-correlation makes sense only if you have an "time-seris like" ordering otherwise it doesn't make a lot of sense. Also, independence => correlation = 0, but not the other way. – suncoolsu Oct 20 '10 at 08:46
  • Why do you want to know if the data are independent? If you are fitting a model the model might assume the **errors** are IID, not the data. Can you provide more context - either as a comment or (better) by editing your Question (see the edit link beneath the tags)? – Gavin Simpson Oct 20 '10 at 08:57

2 Answers2

4

You don't frame the two problems the right way.

Given a random dataset, ie a collection of observations $x_{ij}$ lying in general position you can always make the $n$ $x_{i}\in\mathbb{R}^p$ independent from one another by randomly shuffling the $n$ indexes. The real question is whether you will lose information doing this. In some context you will (times series, panel data, cluster analysis, functional analysis,...) in others you won't. That's for the first I in IID.

The 'ID' is also defined with respect to what you mean by distribution. Any mixture of distribution is also a distribution. Most often, 'ID' is a portmanteau term for 'unimodal'.

user603
  • 21,225
  • 3
  • 71
  • 135
2

Whether a set of observations are iid or not is a decision that is typically taken after a consideration of the underlying data generating process. In your case, the underlying data generating process seems to be the measurements of the speed of a river. I would not consider these observations to be independent. If a particular measurement is on the high end of the scale the next measurement is also likely to be on the high end of the scale. In other words, if I know one measurement I can infer something about the likely values for my next measurement. However, the values are likely to be identically distributed as the errors in your measurement probably come from the methodology used to measure velocity. I would imagine that you would use the same methodology to collect multiple measurements. You could of course use formal statistical tests to see if your observations are iid but for that you have to specify a data generating process, estimate the model under iid assumptions and examine for residuals for deviation from iid.

However, do note that I know nothing about engineering wavelets, wavelet space so I may be way-off in my above assumptions/answer.

  • the velocity components (3D) belong to a turbulent flow. Although the trend follows a sinusoidal pattern, there are high-frequency variations. in my case the trend follows 1/12hr frequency but measurement frequency is 1 Hz, i.e. if at time t speed is 20cm/s, speed at t+1 the speed might have any value saying 20+-5 cm/s. Hence, essentially i cannot predict the next value in a turbulent field. – K-1 Oct 21 '10 at 01:11
  • could you please explain more about checking ID condition? – K-1 Oct 21 '10 at 01:11
  • Well, I do not know enough about your context to give a sensible comment but you have to model the velocity vector as a time series of some sort with an additive error term which you assume is iid. You estimate the model and test if the residuals are iid. Perhaps, you could ask another question with some context of the data and your goals are as far as data analysis is concerned. Perhaps, the issue of iid or not is not that relevant given what you want to achieve? –  Oct 21 '10 at 01:39