0

I'm looking for some methods of variable selection on large datasets.The number of variables are around 30-40, but the number of observations is quite large (around 36000000) Any methods which I know of like stepwise regression or orthogonal matching (or rather their implementation in libraries like scikit-learn) can't really handle such data. The first problem is they always try to fetch all of the data in memory. I didnt get past that to know if there will be more problems.
Does anyone know of any libraries which maybe able to handle such datasets.
Even broader methods for selection is fine ( not necessarily a stepwise regression).
Related to Variable selection in large datasets

(As pointed out by Richard hte terms may cause confusion, hence rewording the question a little)

Community
  • 1
ssj3892414
  • 61
  • 5
  • Do you really need all the data? Why don't use just a sample? – Artem Sobolev Dec 25 '14 at 07:31
  • It would help to disentangle the different aspects of the problem. First, what do you mean by a large number of samples? Is it one large sample (very long time series or very wide cross section) or many copies of (not-so-large) samples? If there are multiple (not-so-large) samples, what trouble does that cause? Can't you handle each sample separately? Second, do stepwise regression and the other methods fail because the way they are implemented they can't handle large amounts of data? I guess the methods are fine in general but a particular implementation might fail. Please be more specific. – Richard Hardy Dec 25 '14 at 12:55
  • I should have been more careful with terminology.By no of samples is large I mean the number of observations (as you said one long time series). @Richard Theoretically I think these methods being not exponentially complex can handle any amount of data, but yeah the problem I faced immediately is memory constraint. Since it didnt procced further can't really say. Depdninf on how inversion is done maybe matrix inversion will be difficult/unstable (not really sure about this). But first concern is really on R or python trying to pull all data into memoey – ssj3892414 Dec 25 '14 at 15:58
  • Now that it is clearer what the actual problem is, I don't think I can help much more. I suggest you highlight the actual bottleneck even more (put in bold or something) so that it is clear for every new reader what it is. You could also indicate more precisely how long your time series are so that we would know better how large a sample we are talking about. – Richard Hardy Dec 25 '14 at 16:45
  • 1
    SAS, if you have it, is usually good at this sort of stuff. SAS has a unique way of processing data (via the program data vector) making it very memory efficient. Otherwise you can still do this in R or similar language (though it requires more thought). In any case it helps to know if your data is stationary and ergodic, or better yet i.i.d. If you don't know you can sample your data to find out. If your data is and you are using R, random sampling may be the easiest way to go, another option may be to program your own regression function that reads only parts of your data at a time. – Zachary Blumenfeld Dec 26 '14 at 03:38
  • 2
    As barmaley.exe said. Just sample it. The first rule of big data is to select a representative sample and work with that. This speeds all the rest of your analysis and investigation. You might therefore consider writing classes for performing the sampling that slot into scikit_learn and therefore allow you to iterate over different samples of your data rather than using single card coded sample. – seanv507 Dec 26 '14 at 09:56
  • You might want to investigate vowpal wabbit which can certainly handle this data size because it works iteratively loading data as it goes – seanv507 Dec 26 '14 at 20:49
  • The thread on building large regression models at http://stats.stackexchange.com/questions/7815 augments the good advice already given by @seanv507. – whuber Dec 28 '14 at 18:18

1 Answers1

1

This may not be an answer you want to see. But I will try.

You have a data set with 40 variables (features) and huge number of observations. You want to select a subset of variables from the 40 variables you have. I guess this is a "supervised" feature selection problem e.g., select subset of variables so as to minimize prediction errors on some variable.

First thing to try is to subsample the observations. Then run your favorite feature selection algorithm. If you cannot subsample (say they are not independent), here are a few things to try. Let k be the number of features you want to have.

  1. Variable ranking. Evaluate each variable independently with a scoring function. Rank and select top k features. This does not take into account feature interaction or feature redundancy. This assumes data slice from considering just one feature fits in memory.
  2. Forward search. Start from an empty set. Greedily add one feature at a time so that some measure (e.g. prediction error) defined on the current set of variables is minimized. Redundancy is taken into account but not feature interaction. This assumes data slice from considering k features fits in memory.

Without further details on your data, I cannot say much. Page 3 of this paper gives a sequence of checklist you might want to check.

wij
  • 1,893
  • 11
  • 18