11

This is my training data: 200,000 Examples x 10,000 Features. So my training data matrix is - 200,000 x 10,000.

I managed to save this in a flat file without having memory issues by saving every dataset one by one (one example after another) as I generate the features for each example.

But, now when I use Milk, SVMlight, or any other Machine Learning Algorithm, everything tries to load the whole training data into memory instead of training it one by one. However I just have 8 GB RAM, so I cannot proceed this way.

Do you know of anyway I could train the algorithm one dataset by one dataset? I.e., so that at any instant I just have one dataset loaded into memory, while training.

madCode
  • 736
  • 1
  • 5
  • 14
  • 1
    What do you mean by "save every dataset one by one?" Do you mean "every line of the data?" If you have multiple datasets, which you save one by one, why not load them one by one and fit the algorithm to each one? – Zach Jul 09 '12 at 17:58
  • 1
    Does "one dataset by one dataset" mean line by line? Ie one dataset = 10000 features? If that is the case, then online algorithms might be something that you are looking for, see: http://en.wikipedia.org/wiki/Online_algorithm and http://en.wikipedia.org/wiki/Online_machine_learning. There exists online version for many machine leaning algorithms, for example SVM and random forests. – Herra Huu Jul 09 '12 at 18:20
  • Thanks.. Zach and Herra. I edited the question to define one-by-one more clearly. And yes, I was thinking of Online Learning but never thought about online algorithms, let me read up on that and try it out. – madCode Jul 09 '12 at 18:54

3 Answers3

4

I believe the term for this type of learning is out-of-core learning. One suggestion is vowpal wabbit, which has a convenient R library, as well as libraries for many other languages.

Zach
  • 22,308
  • 18
  • 114
  • 158
  • I'm having dependencies issues with boost while installing it. do you have any idea on why i get this? http://bit.ly/L939DO – madCode Jul 11 '12 at 16:35
  • @madCode I've never actually used vowpal wabbit, so I can't help you install it. I've heard their mailing list is excellent, and I'm sure you can find help there for setting it up. – Zach Jul 17 '12 at 19:17
  • Hey..Zach. It worked fine. I got it installed and even give me predictions. thanks :-) – madCode Jul 17 '12 at 20:38
1

I heartily second Zach's suggestion. vowpal wabbit is an excellent option, and you'd be suprised by its speed. A 200k by 10k data-set is not considered large by vowpal wabbit's norms.

vowpal_wabbit (available in source form via https://github.com/JohnLangford/vowpal_wabbit, an older version is available as a standard package in Ubuntu universe) is a fast online linear + bilinear learner, with very flexible input. You may mix binary and numeric-valued features. There's no need to number the features as variable names will work "as is". It has a ton of options, algorithms, reductions, loss-functions, and all-in-all great flexibility. You may join the mailing list (find it via github) and ask any question. The community is very knowledgable and supportive.

arielf
  • 1,128
  • 11
  • 15
1

I answered similar question here. Point is most machine learning/data mining algorithms are batch learners that is they load all data to memory. Therefore you need different tools for very large data sets as you have. See that questions's tools also.

Online Learning is a way to reduce memory footprint of algorithms.

Community
  • 1
Atilla Ozgur
  • 1,251
  • 1
  • 11
  • 17