8

I'm trying to train a regression tree with some very large data I have: approx 3Tb.

I'm using scikit-learn and of course there is no way I can load that amount of data on memory. Doing some online research I found that some scikit-learn algorithms have a partial_fit method which can be used for this purpose. Unfortunately scikit-learn decision trees don't have a partial fit method.

I wonder if any of you have come across this problem and if there is an alternative to handle it.

By the way, my data is stored in a pandas data frame.

Ferdi
  • 4,882
  • 7
  • 42
  • 62
Ambesh
  • 303
  • 4
  • 10

1 Answers1

10

Scikit-learn only offers implementations of the most common Decision Tree Algorithms (D3, C4.5, C5.0 and CART). These depend on having the whole dataset in memory, so there is no way to use partial-fit on them. You could only learn multiple decision trees on small subsets of your data and arrange them into a random forest.

There are some other algorithms for inducing Decision Trees that work with on-disk or streaming data. I think the most known algorithms are SLIQ, SPRINT and HOEFFDING Trees (see this paper). While none of them seem to be implemented in any of the common machine learning frameworks (afaik), you can find some stand-alone python implementations on github. And if you are using Spark, they have some Decision Tree algorithms for big data too.

PS: btw, how are you storing 3Tb of data in one pandas dataframe? (Are you using PyTables?)

Community
  • 1
Bobipuegi
  • 753
  • 5
  • 12
  • Hi thanks for the information. Actually what I meant is that I convert the data to pandas from SQL. So the 3Tb are in sequel and I wand to do incremental training with pandas data frames. – Ambesh Sep 26 '17 at 14:14
  • 1
    Okay, then you really might take a look at that paper I linked, because it specifically focuses on streaming data. – Bobipuegi Sep 26 '17 at 14:16
  • 2
    Following the link to that paper eventually led me to find https://scikit-multiflow.github.io/scikit-multiflow/skmultiflow.classification.trees.html which looks promising. – Justin Harris Jul 04 '19 at 17:23